Python code to issue TMCC, Legacy, and PDI Commands

@BillYo414 posted:

Would these events include items outside of the Lionel ecosystem? Like say environmental sounds, scenery animations, or digital information? And as far as digital information, that would be things like moving digital freight around the layout.

Yes, absolutely. For example, if you had a gate man connected to an LCS ASC2, you could trigger him coming out of his little house when the train went over the sensor track. True, you could accomplish the same thing with physical wiring, but there is an elegance to doing so by issuing TMCC commands and even using random numbers to control how long the response action lasted.

@Patrick Keistler posted:

This is starting to smell a lot like @Ryaninspiron's gamification project.

Yup, @Ryaninspiron has done a bunch of programming work... the posts in his profile are worth a peek for sure.

@cdswindell my layout is based on a steel mill. It's still in the very early stages of development but I often plan later stages for future reference since I find slowly working on an idea helps me get it right the first or second time when I build it.

I've toyed with the idea of a fast clock but many people aren't experts on steel making and running against a rigid clock might end up causing chaos in an ops session. I was thinking an event driven sequence would be better. So a car being spotted under the blast furnace would trigger a timer and the furnace tapping animation sequence (lights, smoke, sound) would occur after a set amount of time.

This is probably worth emailing you about for the finer details at this point but having the ability to use the locomotive/car itself to activate this would save me some wiring and hiding of sensors.

I should probably start learning how to use LCS and Python!

Hey all, thank you for the @mentions 😁, I appreciate it. Yes stalled but not completely dead in the water at the PDI phase. I wanted to skip over Wi-Fi because that introduces a whole new level of hassle to the user not to mention possibilities configuration issues getting in the way. For my game idea I wanted to kind of develop the game as a PDI module to make it as easy to implement as possible for the widest number of people. I did take on a new job in May which have been consuming so much more time besides what my business had originally. But I am absolutely down to contribute anything I can to any project hopefully continue with everything else I was doing too. My availability has been starting to get a little better lately (knock on wood).

Most recently I've also been working with my train club TMB to . For them a large scale map of the layout on the wall with live tracking block occupancy across the layout. Which as of last year is now located in a 6,000 ft basement below a church.

I'm going to have to do some reading to catch up on these last two pages in this thread. But I love any and all projects that give LCS a purpose for existing, besides setup and simple location pinging in the old lcs ipad app. Plus I love to see interaction happening with the layout I think that's where the next era of fun to be had can be found.

Plus I recently learned some fun tricks using python code that could absolutely help me break down the PDI data, struct.pack and struct.unpack. Part of what made things tricky before was working in Arduino code exclusively and facing off with decoding raw binary data via bitmasking was not nearly as fun.

And here are a few of the modules I was tinkering with to develop these interfaces. On the back left corner there is a radio that is supposedly capable of impersonating a DCS remote which I was planning to interface with TMCC.

@Ryaninspiron posted:

Plus I recently learned some fun tricks using python code that could absolutely help me break down the PDI data, struct.pack and struct.unpack. Part of what made things tricky before was working in Arduino code exclusively and facing off with decoding raw binary data via bitmasking was not nearly as fun.

@Ryaninspiron, we may make a good team I like decoding raw binary data via bitmasking !

But seriously, I have code in place that decodes 80 to 90% of the PDI commands in the confidential Lionel PDI/LCS docs. The code is all checked in on GitHub. Just search for the PyLegacy project!

  — Dave

I decided to start maintaining state for the LCS Sensor Track. It plugged into the stack with about 90 minutes of coding, which felt pretty good. I could also manage state for (the) other LCS modules, but I’m not sure I’ll need to. I was already processing state changes signaled by the Base, BPC2, ACS2, and STM2 by translating their updates into Switch, Accessory, and Engine/Train space (e.g., recording an STM2 update as a Switch Thrown update).

It will now be easy to have an engine/train go to slow speed or go to normal speed when it crosses over a Sensor Track programmed to generate that sequence.

  — Dave

@Ryaninspiron posted:

Plus I recently learned some fun tricks using python code that could absolutely help me break down the PDI data, struct.pack and struct.unpack. Part of what made things tricky before was working in Arduino code exclusively and facing off with decoding raw binary data via bitmasking was not nearly as fun.

@Ryaninspiron, huh, I was unaware of the struct package! That would be very useful for my code that writes PDI/LCS packets that I send to the base, thanks!

  — Dave

@BillYo414 posted:

@cdswindell my layout is based on a steel mill. It's still in the very early stages of development but I often plan later stages for future reference since I find slowly working on an idea helps me get it right the first or second time when I build it.

I've toyed with the idea of a fast clock but many people aren't experts on steel making and running against a rigid clock might end up causing chaos in an ops session. I was thinking an event driven sequence would be better. So a car being spotted under the blast furnace would trigger a timer and the furnace tapping animation sequence (lights, smoke, sound) would occur after a set amount of time.

This is probably worth emailing you about for the finer details at this point but having the ability to use the locomotive/car itself to activate this would save me some wiring and hiding of sensors.

I should probably start learning how to use LCS and Python!

@BillYo414, happy to have an email conversation. The code in its current form can definitely generate events in response to an engine (or car with an IR transmitter) traveling over a sensor track. The event packet contains the ID of the activated sensor track, the direction of travel, the engine/train ID passing over, and a bunch of what I call simulated state (fuel/water level, odometer, total runtime, etc.)

You could write code that “subscribes” to sensor track events. That code would then receive these updates and could go from there. You can also access current engine speed and direction from other components of my system and/or by subscribing to engine/train events.

I don’t currently have any code to sense engine position (other than the sensor track), so I’m not sure how you would tell when a car was next to/underneath your steel mill stuff. My code gives you fine control over train speed (absolute speed step control), as well as commands to immediately stop engine motion immediately. In fact, I’ve implemented all of the TMCC commands in the public Lionel spec (except those to build/tear down train consists)!

  — Dave  

@cdswindell posted:

@Ryaninspiron, we may make a good team I like decoding raw binary data via bitmasking !

But seriously, I have code in place that decodes 80 to 90% of the PDI commands in the confidential Lionel PDI/LCS docs. The code is all checked in on GitHub. Just search for the PyLegacy project!

  — Dave

Absolutely would be great to join forces, and since you love bitmasking so much you might just love to see the work I've done on reverse engineering the sensor box cars and therefore the ability to create postage stamp size modules to add sensor track capabilities at any number of cars or engines. I've already demonstrated one of these modules in operation on one of my YouTube videos on the subject. But until now I haven't had anybody else to share the load with. I had actually ordered about 20 old school irda transceivers, and use that to capture the data into a bit array that I then saved into Arduino code, and then put matching code to retransmit that bit stream back out on the two in IR LEDs. I also did testing and figured out exactly which wavelength of ir LED you can use off the shelf from Amazon as a sensor card transmitter. I have one other person who tested and deployed my code successfully too. But they did not end up having time to help me make progress with completely cracking the code such as transmitting a-la-carte engine IDs and names. But I do have a demo on youtube from several years ago where I trigger sounds on a line Chief engine using my home made sensor car. https://youtu.be/qgGiVOycX8o?si=cXwrYXJELr2Zrtvv

So yes I agree we could make an amazing team, I'm also an LCS partner too. Honestly that made me nervous about what I was allowed to disclose and it made me hesitant to simply publish my code before.

Of course everything I learned on the sensor car box car was through traditional reverse engineering since none of that documentation was included in the partner program. I also had set up a discord server to host discussions about this sensor car reverse engineering project including links to Amazon for buying all the hardware to test it on your own. https://discord.gg/ptmd79Zd

@cdswindell posted:

@BillYo414, happy to have an email conversation. The code in its current form can definitely generate events in response to an engine (or car with an IR transmitter) traveling over a sensor track. The event packet contains the ID of the activated sensor track, the direction of travel, the engine/train ID passing over, and a bunch of what I call simulated state (fuel/water level, odometer, total runtime, etc.)

You could write code that “subscribes” to sensor track events. That code would then receive these updates and could go from there. You can also access current engine speed and direction from other components of my system and/or by subscribing to engine/train events.

I don’t currently have any code to sense engine position (other than the sensor track), so I’m not sure how you would tell when a car was next to/underneath your steel mill stuff. My code gives you fine control over train speed (absolute speed step control), as well as commands to immediately stop engine motion immediately. In fact, I’ve implemented all of the TMCC commands in the public Lionel spec (except those to build/tear down train consists)!

  — Dave  

Wow I was just in the middle of implementing a project that uses industrial communication modules(modbus rtu) RS485 to capture the occupancy of all the blocks on a large scale 6,000 sqft layout and was just envisioning tracking engine IDs along the way by tacking them onto the continuous chain of block occupancy that passes over a sensor track.

This feels a bit like a milestone to me. I now have trains automatically slowing down or speeding up as they pass over a sensor track! They only do this if the sensor track sequence is set to one of the two values where a speed dialogue is given, based on the direction of travel. I have some work to do to make the speed up or slow down, look more realistic by taking the train’s Momentum, setting and current speed into account, but I have all the pieces sitting around to do this (in code, of course).

I’ll post a video to YouTube and include its link in a day or two.

I still haven’t started packaging.

  — Dave

Dave,

Looks like I'm in... with a  little overkill.  I just pulled the trigger on the 8G Pi 5 (bare board).  I have a Samsung M.2 970 Pro that I just took out of the workstation and it's gathering dust so I ordered the Pineboards POE+ (2280 M.2).  Not the cheapest configuration... but hey, it's for the trains.

@Dennis-LaRock posted:

Dave,

Looks like I'm in... with a  little overkill.  I just pulled the trigger on the 8G Pi 5 (bare board).  I have a Samsung M.2 970 Pro that I just took out of the workstation and it's gathering dust so I ordered the Pineboards POE+ (2280 M.2).  Not the cheapest configuration... but hey, it's for the trains.

That will more than do!!! FYI, I just installed the M2 hat that Raspberry Pi itself distributes and it worked perfectly. One thing to watch out for. I had better luck using the “Copy SD card” in the Pi gui to clone my system image onto the M2 drive (requires a VNC connection; everything else I do via ssh).

  — Dave

Pi5 in the house... didn't use the micro sd drive, booted straight off the USB with a Samsung T5 ssd.

Just checked in support to ramp speeds up and down, more like the Cab2 and Cab3 throttles do. You can still jump immediately to a desired speed, but the default is to now use a ramp that also takes the engine’s momentum into account.

ill post a video tomorrow. Looks nice when the speed is changed by going over a sensor track!!

  — Dave

Checked in support for a Rotary Encoder. Think of these like digital potentiometers, except they rotate round-and-round, just like the big red throttle knob on the Cab 2.

I've incorporated this support into GpioHandler.gantry_crane and GpioHandler.track_crane to rotate the cab. Of course, with that said, when I unboxed my Lionel Legacy CSX Track Crane car, I found it was missing a gear necessary to rotate the cab! But my code does send the right commands and I can see the gears they did install turn in the correct direction. Sigh

Here's what the code looks like a user would have to write to build a dedicated panel to control a track crane car:

GpioHandler.crane_car(
    address=91,
    cab_pin1=20,
    cab_pin2=21,
    adc_channel=0, (1)
    boom_pin=16,
    bh_pin=25,
    sh_pin=26,
    boom_led_pin=23, (2)
    bh_led_pin=24, (2)
    sh_led_pin=22 (2)
)

I better get going on this before you do everything cool!

I really can’t wait to build my own legacy controller - I just keep getting sidetracked with other projects!!!

@cdswindell posted:

…use a ramp that also takes the engine’s momentum into account

  — Dave

I was playing with “momentum” in analog controllers over the summer. I found that for it to really look and feel right, you need to be playing calculus games - actively changing the rate of change over time (second derivative? I forget, it’s been an awfully long time…) to ease in and out of acceleration and deceleration. Otherwise it’s just like a sort of delayed speed change effect.  With complete control over the software stack like you are building, I bet you can really do momentum right!

@christie posted:

I was playing with “momentum” in analog controllers over the summer. I found that for it to really look and feel right, you need to be playing calculus games - actively changing the rate of change over time (second derivative? I forget, it’s been an awfully long time…) to ease in and out of acceleration and deceleration. Otherwise it’s just like a sort of delayed speed change effect.  With complete control over the software stack like you are building, I bet you can really do momentum right!

Calculus was an awfully long time ago for me, but you absolutely could build any kind of speed table that you want. I took a look at what the Lionel Cab3 controller does and tried to emulate that. My implementation at this point is all based on linear map, but I guess I was just lazy.

  -- Dave

@christie posted:

I better get going on this before you do everything cool!

I really can’t wait to build my own legacy controller - I just keep getting sidetracked with other projects!!!

The software I've developed implements ALL of the TMCC commands and most of the PDI/LCS commands. you absolutely could!

  -- Dave

Added a "decode" command to PyTrain to convert hex codes seen in the LCS Monitor app (or Wireshark Sniffer) to the corresponding TMCC or LCS Command.

For example:

>> decode D1 28 F8 87 CC 8D DF
0xd128f887cc8ddf --> [PDI Tmcc_Rx [ENGINE 67 ENGINE_LABOR 12 (0xf887cc)] (0x28f887cc)]
>> decode 0xF88703
0xf88703 --> [ENGINE 67 REVERSE_DIRECTION (0xf88703)]

-- Dave

Finally added logging to the code. Log messages can go to both a file and the console at the same time. Logging can also capture command echoing, so you can have a complex transcript of all the TMCC commands happening on your layout, be they from a Cab 2, a Cab 3, or from the PyTrain command line. This really lets you see the commands the Cab 2 and Cab 3 generate when you operate a control; you’d be surprised!

  — Dave

I received my Gantry Crane today and ****ed if the components I’ve wired up to control it did just that! I turned the rotary encoder and the crane cab turned. I pushed up and down on the joystick and the magnet raised and lowered. And I pushed left and right on the joystick and the crane moved left and right!!

I have some tweaking to do, as the code that sends the numeric prefix (<Aux 1> [1, 2, 3]) needs to be synchronized so that state isn’t lost when you operate multiple controls at the same time, but it works!

  — Dave

I've switched gears to working on support for making the system easier to use and more bulletproof. This is involved two changes. The first is I've developed some scripts that allow you to run the software when the raspberry pie boots up. This will be important because when powers turned onto the layout, I want the pies that are powering the control panels to just boot up and be working, like an embedded device.

The second change is to use a discovery service to allow the PyTrain server(s) to register their presence on the local network. This will allow the client applications to not need to fool with IP addresses. They will auto-discover The IP address of the server, and just simply connect to it. Since the Base 3 Doesn't echo out very many of the commands it receives in, I still think it will be advantageous to have an LCS Ser2 connected to the server. That said, I'm investigating if it makes sense to have each client send train commands directly to the Base 3 over WiFi. This should help distribute the load.

I'm working now on actually building the first control panel to operate my gantry crane. I'll send pictures along once there's something to show. By the way, all of the code is checked in.

  -- Dave

Added support to allow engines to be looked up by road number. This makes it easier to address engines, as the need to remember the associated TMCC IDs is eliminated. This is a precursor to adding support for 4 digit TMCC IDs…

I also figured out how to update the Base 3 and Cab 2 in a more efficient way. This support isn’t checked in yet, but will come online this week.

i also am about to fabricate my first controller panel! It will just be a prototype. I’ve switched over to using an I2C matrix keypad, which should give me 19 GPIO pins to wire up to fire commands!

  — Dave

Here's a picture of the completed prototype:

The throttle knob is a $12 rotary encoder intended for CNC machines. It has a nice feel with distinct "clicks" as you rotate. My attempt at soldering leads onto a "soft pot" was a total failure. I hadn't noticed you could buy them with connectors already attached. That Amazon order arrives next week

On the software side, I've added a number of commands intended to make operating a cluster of PyTrain servers (actually, 1 server and many clients). You can now issue a single command that will update the PyTrain software across the cluster, as well as upgrade the Linux OS on all systems with one command from one system. I've also enabled the ability to run a client on the same system as a server. Why, you may ask? My intension is to start my software automatically when each box boots. This way, when you turn the power on to your layout (and control panels), all of the integrated Pi systems will just boot up and run the code. But with this method, where all the systems are "headless", there is no operator console available to issue commands! This is where the need to run a client on the server comes in. When it comes time to upgrade the OS or my code, you cans onto one of the systems (preferably the primary server), run the software in "client" mode, then issue the appropriate command (update to update my software, upgrade to upgrade the raspberry pi OS as well as my software), or reboot, because sometimes...

I also added an ability to automatically find the Base 3 on a local network. I used Wireshark to try to figure out how the Cab 3 finds the Base, and, as best as I can tell, it simply tries all IP addresses on the local network (255 - 3, or 252), stopping once it succeeds. I am using Python's "multiprocessor support" to do this, so on my 24 core development Mac, it finds the base in just a few seconds! On a Pi 5, with just 4 cores, it can take 10 - 20 seconds to find the Base, but as it only needs to do this once on start-up, and only on the "server" instance (clients can get the base address from the server), this may be sufficient. I will probably hard-wire the base address into my server start-up script, but I wanted the option...

  -- Dave

For 30 years we've been stuck with the control surfaces that Lionel gave us or those developed/licensed by a small number of competitors and developers. Now, we can create basically any interface our imagination can think up, or anything our layouts demand. This is amazing work, Dave.

This is awesome work.  Well done, Dave.

Now that they're officially out of the handheld controller game, I hope Lionel takes notice and really open things up with Legacy to allow 3rd parties to bring their own control hardware, and even actively support integration with Base3 and a lot of the other stuff they make.  This is a great way to get technically minded folks into the hobby, too.  I don't really see how it could hurt them unless they ever change their mind about making hardware remotes, but even then, most people would probably just buy theirs again anyway.

Just added an "when_pressed_or_held" handler that can issue one of 2 commands, based on how long a button is held. This is useful for handling things like startup and shutdown, where there are really two flavors of the command; one that does the action without dialog, and another that does it with dialog. The threshold is settable, but .3 seconds seems to work nicely...

  -- Dave

Just added more realistic syncing of Diesel RPM levels as you increase/decrease throttle via the PyTrain methods. I have to do the same for Engine Labor as well (TBD).

Does anyone know offhand if Engine Labor (effects) applies to Steam and/or Electric, as it does to Diesel? I'm only applying RPM effects to Diesel engines (as does the Cab 3).

  -- Dave

I do believe they apply to steam, you use the same controls on the CAB1 to increase the labor for steam.  I can't say for sure how that translates into the Legacy commands, but given you use the same techniques for steam or diesel to increase labor, I would suspect it would be the same commands.

@gunrunnerjohn posted:

I do believe they apply to steam, you use the same controls on the CAB1 to increase the labor for steam.  I can't say for sure how that translates into the Legacy commands, but given you use the same techniques for steam or diesel to increase labor, I would suspect it would be the same commands.

Thanks John.

Yes, the commands are the same, I just didn't know if the concept applied to steam. I would have thought the concept of RPM applied to steam too, but if you pull up a steam engine on the Cab 3, the RPM bars to the left of the throttle disappear...

It's not RPM on steam, it's labor.  Obviously, unless the steam engine changes speed suddenly or has wheel-slip, the RPM won't change.  However, if you pull the train brake down and then run the speed up past where the train brake is limiting the speed, the labor increases.  By the same token, if you have momentum set at high and spin the throttle up quickly, you'll also get increased labor until the actual locomotive speed catches up with the speed setting.

The concept of "working harder" applies, but with a diesel, it's the prime mover ramping up in speed to generate more power, with steam it's the amount of steam fed to the cylinders to produce more power to the wheels.  Same idea, just a totally different way of accomplishing it.

@cdswindell posted:

Thanks John.

Yes, the commands are the same, I just didn't know if the concept applied to steam. I would have thought the concept of RPM applied to steam too, but if you pull up a steam engine on the Cab 3, the RPM bars to the left of the throttle disappear...

I think you’re taking about two different things. RMP definitely is only in diesels, but going into the speed menu on the Cab-2 and now you have labor adjustments labeled EFX. This shows up on diesel, steam, and I’m pretty sure it’s also on electrics.

I haven’t looked for that menu on the Cab3 App. I’ll get back to the base 3 tonight and see if I can find it.

Finally completed support for using I2C digital I/O extender boards to add more buttons. The code creates a new type of gpiozero “Button” that has all of the same behavior. This means that you can use the “when_pressed” and “when_held” call backs, which was central to my design.

The code was developed for the CQ Robots Ocean 16 port board, but should work with any board with a MCP23017 chip. The nice thing about this chip is it provides interrupt support so you do not have to poll the board looking for button presses.

I don’t think I’m going to take the time now to add support to drive LEDs, although I think doing so would be pretty simple.

  — Dave

@cdswindell posted:

Finally completed support for using I2C digital I/O extender boards to add more buttons. The code creates a new type of gpiozero “Button” that has all of the same behavior. This means that you can use the “when_pressed” and “when_held” call backs, which was central to my design.

The code was developed for the CQ Robots Ocean 16 port board, but should work with any board with a MCP23017 chip. The nice thing about this chip is it provides interrupt support so you do not have to poll the board looking for button presses.

I don’t think I’m going to take the time now to add support to drive LEDs, although I think doing so would be pretty simple.

  — Dave

It was pretty simple; support for I2C LEDs is now in place

Code has been modified to use gpio extender ports or regular pi ports in all GpioHandler methods (route, switch, controller, etc).

  — Dave

I finally tackled packaging and now have the code up on Pypi’s test site. And, as I had hoped, you will soon be able to get up and running by typing 3 commands! I’ve also developed a tool to configure a Raspberry Pi to run the code and remove extraneous services and packages not needed to run PyTrain. This helps reduce the system load on the small Pi devices, although it runs fine even without the optimizations.

This wraps up the development efforts for PyTrain. The major outstanding task is completing four digit addressing support. I’m hoping to have a four digit addressable engine next month, at which point I will complete that support.

Next up, I've decided to implement a Restful API. This will allow developers working in other languages to control trains. And it is a necessary component for my next major feature; Alexa Control! With the ability to query ngine names and road numbers, and access to the complete command library, this should allow me to develop a pretty full-featured Alexa skill to run my layout.

  -- Dave