DAN][ Controller with Ethernet and FTP Server

14 posts / 0 new
Last post
MacFly's picture
Offline
Last seen: 8 hours 1 min ago
Joined: Nov 7 2019 - 13:49
Posts: 484
DAN][ Controller with Ethernet and FTP Server

I haven't seen reports of people using the DAN][ controller's Ethernet option. This is also a neat feature that Daniel added to the card. He added support to "IP65", the 6502's TCP/IP library. There is an example disk in the github project.

I now added something else: I squeezed a tiny FTP server into the Dan][ controller's "Arduino" firmware itself. It provides network access to the SD cards, so you can remotely upload or download volume images. So this does not run on the Apple II/6502, but completely independently on the tiny Arduino CPU. It has some limitations, but turned out to be much better than I initially expected. I find this super useful, especially if you're developing/working on something and require frequent updates and are tired with messing with these tiny MicroSD cards.

How to install

You need a WIZnet 5500 module. They are cheap. 8-10 Euro/bucks a piece. And you need a 10-pin ribbon cable with IDC connectors to connect it to the "J6" header:

I installed mine in one of the DB19 breakouts of the Apple IIe case, using a 3D printed bracket. The DB19 breakouts are too narrow for the Ethernet port, but it works in this way:

You can leave out/remove the two resistors R27 and R28 on the controller PCB when you connect the WIZnet. These connect signals to the header which are not meant for the WIZnet, but for an external SD card adapter. Nothing will break if you have these soldered, but they unnecessarily connect outputs to ground via those 470 Ohm resistors:

You need new firmware for the Arduino, if you want the FTP server option (see github below). Only the Arduino needs an update. The card's (E)EPROM is unrelated to this feature.

The FTP server uses a static IP address - no DHCP autoconfig etc. You can change the default IP address in the sources, or use a simple utility to configure the controller. I made a 140K ProDOS "ipconfig" disk image (see "dsk" folder), which you can write to a disk - or, of course, store it as a volume image on the DAN II drive itself, using the .po image file. Configuration should be self explanatory. It also allows to change the MAC address. But you only need to worry about MACs when you are as crazy as me, and have multiple of these cards in your network. Then the MAC addresses must be configured to be unique:

With the new firmware, the FTP server is automatically active when the WIZnet is present - and an Ethernet cable is plugged. The FTP server is accessible within 2-3 seconds after resetting the Apple II. You will see the LED "D11", which is just above to the pin header, faintly flashing. Once you connect via FTP, this LED will light bright and (almost) continuously, since Arduino and WIZnet are talking non-stop.

Limitations

The tiny FTP server comes with some limitations - mainly due to limited space:

  • You can only "see" the "BLKDEVxx.PO" files for the volume images via FTP. You cannot see or access any other files on the SD cards.
  • You cannot navigate or create new folders. It only provides fixed folders: the root level and two subdirectories "SD1" and "SD2" for the SD-cards. The SD card directories contain the the respective BLKDEVxx.PO files. That's it.
  • You cannot create, remove or rename files. Yes, renaming would be handy to organize volumes on the SD cards. But, no, it's not supported.
  • Literally, the only thing you can do is read or write to any of the existing BLKDEVxx.PO volume files.

Preparing the SD cards

You should prepare new SD cards by writing (dummy) images for each BLKDEVxx.PO file before using the FTP server. Use any file of the largest supported size (65535 blocks of 512 bytes = 33553920 bytes or about 33MB). Or use any larger file - doesn't matter. Just important that the files are allocated with sufficient size, so they become usable for remote access - and you never need to remove or manually touch the SD cards again. The Arduino will never change any disk meta data (directory entries, file allocation tables etc). This is good, since it basically excludes any chance that the disks could ever get corrupted. The Arduino will only ever update the raw data blocks of existing files - nothing else.

Hints on using Apple II and the FTP server

When an FTP session is connected, the Apple II's access is suspended. The Apple II/ProDOS will hang if it tries to access the DAN][ controller. But, no worries, once you disconnect your FTP session, the Apple II immediately resumes.

Of course, Daniel's original Ethernet support for IP65 also still works. Surprisingly everything still fits into the Arduino flash. When you run a program on the Apple II with the IP65 stack, it takes (remote) control of the Ethernet port. The FTP server on the Arduino is then automatically shut down. So both options can be used - just not at the same time. The FTP server is reenabled by resetting the Apple II.

Remember, resetting the Apple II also resets the Arduino. Don't press Ctrl-RESET while your actively up- or downloading volume images via FTP.

Performance

FTP data transfers at about 170-200KB/s. Transmitting a full 33MB volume image takes a bit less than 3 minutes. Yes, not too fast. But I was actually expecting much worse. And considering the alternative of removing, reinstalling and messing with these tiny MicroSD cards, it's not that bad.

And concerning speed: for frequent image updates, for example when developing/testing, you can always use smaller volume images. Volumes  of any (smaller) size are supported - even if the "BLKDEVxx.PO" files were originally initialized with much larger files.

A standard floppy disk-sized ProDOS image of 140K transfers in less than a second - and you're ready to go (reset and boot the image). I find this really cool: I added a call to a command-line ftp utility to the build makefiles of my projects. You run "make", it compiles/assembles the program, builds a small ProDOS volume image, immediately downloads it via FTP to the DAN ][ controller - all in matter of seconds. You reset the Apple II and - boom! - it runs. Loading an image to the real hardware is almost faster now than launching it in a simulator.

Extras

Though the FTP server is tiny, it has some extras - it is "ProDOS-aware":

  • When you list the files of the SD cards, pay attention to the colums normally showing "file owner" and "group" information. The server uses these fields to report the ProDOS volume name instead. So you can directly see in your FTP client what each "BLKDEV" file actually contains.
  • The server also considers the size of the ProDOS volume. So, when you upload, say an 140K image - but the BLKDEV file in the SD card's DOS file system is larger (33MB), then the FTP server properly shows and treats this as a 140KB file (or whatever volume size is used).

FTP Clients

Basic FTP utilities should all work out of the box. Powerful tools, like FileZilla, also work, but require a configuration setting:

  • The FTP server only supports a single connection. FileZilla always tries to use multiple connections (to navigate directory structures, for file transfers etc). Look at the "connection properties": there is an option "Limit number of simultaneous connections". Check this option. And set the "maximum number of connections" to "1". That's all.

How it looks

So, here's how it looks, for example, when connecting with FileZilla FTP. Notice the sneaky abuse of the "owner/group" file information:

And that's with a bare metal command-line FTP session:

You can find the firmware with FTP server in my experimental "ftp" github branch:

https://github.com/ThorstenBr/Apple2Card/tree/ftp

This also includes my previous boot menu updates. I've used it for a bit now and it seems stable to me, but if you noticed anything, let me know. The "dsk" folder contains the volume images for Daniel's "IP65" examples and for the simple "FTP ipconfig" utility.

 

That's it. Good luck! :)

 

(PS: Bonus question for Apple II fans. The tiny FTP server has no "date/time" support, so reports all files with a fixed date: June 10th, 1977. You know this date, right? :) )

CVT
CVT's picture
Offline
Last seen: 18 hours 24 min ago
Joined: Aug 9 2022 - 00:48
Posts: 1170
This is pretty awesome! Does

This is pretty awesome! Does it work in ad-hock mode? (No router, just an ethernet cable between the Apple II and a laptop.)

MacFly's picture
Offline
Last seen: 8 hours 1 min ago
Joined: Nov 7 2019 - 13:49
Posts: 484
CVT wrote:Does it work in ad
CVT wrote:

Does it work in ad-hock mode? (No router, just an ethernet cable between the Apple II and a laptop.)

 

Yes, it does. Of course you also have manually configure the IP address of the laptop's network port then, since the Arduino neither acts as a DHCP server. :) But otherwise this works fine.

Offline
Last seen: 1 week 3 days ago
Joined: Jun 12 2022 - 23:35
Posts: 111
This is really great work.  I

This is really great work.  I'll have to check it out.  I did not consider this possibility and I am more glad than ever to make the card open source so all kinds of improvements can be made. 

 

By the way, the SDA/SCL is broken out on the card.  It would be possible to connect something like a DS3231 or other I2C devices to it and report the time back to ProDOS.

 

 

CVT
CVT's picture
Offline
Last seen: 18 hours 24 min ago
Joined: Aug 9 2022 - 00:48
Posts: 1170
MacFly wrote:Yes, it does. Of
MacFly wrote:
Yes, it does. Of course you also have manually configure the IP address of the laptop's network port then, since the Arduino neither acts as a DHCP server. :) But otherwise this works fine.

 

Great! For me ad-hoc is really convenient, because all my routers are far away, yet I have an ethernet port on my laptop that I never use, which can stay configured for ad-hoc with a static IP address.

 

I ordered one of these: https://www.aliexpress.com/item/32435030006.html 

MacFly's picture
Offline
Last seen: 8 hours 1 min ago
Joined: Nov 7 2019 - 13:49
Posts: 484
SPI weirdness

When I implemented the FTP/WIZnet feature, I ran into one weird issue which I couldn't explain. The Ethernet driver would regularly freak out when data was written at higher bandwidth to the SD cards. This would somehow cause the connection between the Arduino and WIZnet to completely stall. No data could be sent/received, Ethernet connections were reported as "disconnected", no new connections could be accepted, nothing was working. SD card access, however, continued to work fine.

It would only happen when writing data to the SD cards. Reading was not an issue. I tried different things, like fixed delays. Eventually I added a kludge to the MMC driver which "fixed" the issue reliably: when a block of data is transmitted, the driver now blocks and waits until the SD card is done with its internal operation (flashing the data). This should normally not be necessary. The CPU should be able to do other stuff while an SD card is busy internally. Especially it should be possible to already receive the next block of data via SPI from the WIZnet module, while the SD card is internally completing the flashing operation. So the kludge reduces performance. I added a switch, so the workaround only affects FTP, but not the performance when the Apple II itself writes to the SD cards.

It works fine. I had no more issues afterwards. And FTP still achieves about 200KB/s. But it's weird and the effect is not really explained. My theory was that the SD cards are somehow still interfering with the SPI bus. Maybe something to do with the 3.3V resistor divider, so the SD card's CS lines aren't releasing the bus fast enough. Maybe this was only now causing issues, when quickly switching between different SPI devices (SD card vs WIZnet). But it's weird, why would that only affect write operations, if this was a general issue. And it's just a theory. Could be something else I'm missing and also be a simple SW issue.

Maybe someone has an idea. You can see the kludge I'm talking about in this git commit - around lines 260:

https://github.com/ThorstenBr/Apple2Card/commit/36b499754f9704bd2eb2f1f1db13278437ad40e7

 

CVT
CVT's picture
Offline
Last seen: 18 hours 24 min ago
Joined: Aug 9 2022 - 00:48
Posts: 1170
MacFly wrote:Maybe something
MacFly wrote:

Maybe something to do with the 3.3V resistor divider, so the SD card's CS lines aren't releasing the bus fast enough.

 

Yeah, a 4.7k/10k divider feels very soft. Maybe replace with 470/1k and disconnect the D1/D9 LED or R24/R26. The ATMEGA 328P can drive voltage dividers that are way harder: I went with 80/160 for the R9/R15 divider in order to get the SD slot extenders to work.

MacFly's picture
Offline
Last seen: 8 hours 1 min ago
Joined: Nov 7 2019 - 13:49
Posts: 484
CVT wrote:Yeah, a 4.7k/10k
CVT wrote:

Yeah, a 4.7k/10k divider feels very soft.

 

The mystery is solved. Fortunately the hardware is fine. It's a protocol/SW issue. The SD specification indeed says a card keeps forcing the SPI data line low while it is busy (programming), no matter what the CS line does. The MMC driver takes care of that. But now we're concurrently using the Ethernet SPI driver, which doesn't expect the SPI to be blocked by something else. I've added a slightly better fix. Unfortunately means there's not going to be any significant performance gain...

MacFly's picture
Offline
Last seen: 8 hours 1 min ago
Joined: Nov 7 2019 - 13:49
Posts: 484
More improvements...

I hadn't explicitly mentioned this in my original post - but as some noticed, FTP access was only implemented for SD cards using the FAT filesystem to store ProDOS volume files. It wasn't implemented for SD cards using "raw block mode".

There is a new version now. It adds FTP support for SD cards with "raw block" volumes. Via FTP those now look exactly like "normal" FAT disks. So FTP shows files "BLKDEV00.PO" - "BLKDEV0F.PO", even though technically no such filenames exist on the disk, as volume data is stored in raw blocks. The only way to tell a difference between the two formats is the displayed volume type, visible as a property of the FTP directory listing:

This is a larger update. For safety reasons the firmware now automatically detects the SD card filesystem (FAT vs RAW) - so there is no longer a need to select "0" for RAW mode (while "1"-"9"/"A"-"F" were reserved for FAT disks). This was necessary since otherwise it was too easy to corrupt FAT disks (by selecting "0" for "raw ode" in the menu and then uploading something via FTP - which would have immediately destroyed the filesystem of a FAT card). The format being independent of any manual menu selection also has the benefit that the primary boot volume is now also configurable for raw block cards. The volumes of both card types are now visible in the same way in the boot menu. Each disk type now supports 16 volumes  - and any volume can be selected as the primary volume for each disk (also for RAW mode, where no volume selection was possible before). And for FAT disks there are also 16 volumes now (0-F), since the "0" is no longer connected to the "raw block" selection.

So the features are now almost indentical, no matter which format is used to store the volumes. The remaining difference is that raw block mode also supports access to multiple volumes (using Daniel's "ALLVOLS" utility for ProDOS 8). That feature is not supported for volumes on FAT disks (not yet :) ).

There are a few other minor improvements. Like the IP configuration for FTP, which is now integrated into the boot menu. Configuration no longer requires a separate utility/disk.

Those who installed the previous version should be able to just run the firmware update disk to get the newer firmware onto the "Arduino". Otherwise, upload the hex file with a programmer as before. I packed a ZIP file with all the binaries/disks. Sources also in github, of course.

https://github.com/ThorstenBr/Apple2Card/releases/tag/v2.1.0

CheshireNoir's picture
Offline
Last seen: 6 days 14 hours ago
Joined: Dec 28 2019 - 05:20
Posts: 184
Amazing work!

I think I need to get me one of these cards :-)

 

Chesh

MacFly's picture
Offline
Last seen: 8 hours 1 min ago
Joined: Nov 7 2019 - 13:49
Posts: 484
ADTPro Ethernet Support for the DAN][ Controller

I made a custom build of ADTPro, which supports the DAN][ controller's Ethernet port (Wiznet 5500 module). Another useful feature for the DAN][ card...

You can now upload/download volume images through the card's Ethernet interface using ADTPro. This is mainly useful for floppy images and volumes which are stored on other drives, like external Floppy drives or FloppyEmu. Well, of course you could also use ADTPro to upload or download harddisk volumes to the DAN][ controller's SD cards itself. However, that is much slower and less convenient than using the built-in FTP support (see above): FTP provides immediate access to all volumes stored on the SD cards, while ADTPro can only access the currently mapped volume(s). Also ADTPro is much slower, since all network data is downloaded to the 6502 CPU and would then be written back again to the DANII card (while FTP provides direct access to the SD cards, without involving the 6502).

All credit goes to Dan - and to ADTPro. ADTPro has Ethernet support based on the IP65 library as a standard feature - normally supporting the Uthernet and LANceGS cards. Dan had made a custom variant of IP65 library supporting the DAN][ card. So all I really needed to do was plug it all together and rebuild ADTPro with the custom library.

Only a few extra tweaks were necessary to make it work. The main issue was that ADTPro always automatically probes the slots for supported cards - even before you can reach the manual Ethernet/slot configuration. And probing of unsupported slots wasn't supported with the custom IP65 variant - so it would always lock up. But it's working now.

 

Starting the custom ADTPro disk shows an "Ethernet: DANII Controller" option:

After selecting the Ethernet option, you can run the manual configuration. The correct slot of the DAN][ card should already be auto-detected. But you will need to configure the IP address of your "server" (= your host PC/Mac). But there is nothing special about the configuration - the standard ADTPro manual applies...

I uploaded the adapted ADTPro disk images to GitHub. It's based on ADTPro release 2.1.0. I adapted the version to "2.1.D2" (2.1.0 for DANII), so this custom build isn't confused with standard ADTPro builds. Only the Apple disk image differs. The ADTPro host utility is unchanged:

https://github.com/ThorstenBr/adtproDAN2/releases/tag/v_2_1_D2

 

magnusfalkirk's picture
Offline
Last seen: 14 hours 33 min ago
Joined: Feb 18 2007 - 11:46
Posts: 335
ADTPro Ethernet Support for the DAN][ Controller

MacFly,

Thank you for this and everything else that you have done to help make the Dan][ card even better. Granted Dan built this usefulness into the card from the beginning but that I know of you're the first person to do something to make it easy to use and easy to update.

Offline
Last seen: 1 month 4 weeks ago
Joined: Jul 5 2018 - 09:44
Posts: 2587
magnusfalkirk wrote:MacFly
magnusfalkirk wrote:

MacFly,

Thank you for this and everything else that you have done to help make the Dan][ card even better. Granted Dan built this usefulness into the card from the beginning but that I know of you're the first person to do something to make it easy to use and easy to update.

 

 

Yes, this kind of cooperation really benefits the whole community!  It's awesome!

 

Both Dan and McFly deserve a lot of respect for creating some cool hardware and making it even better!

 

 

 

Offline
Last seen: 1 week 3 days ago
Joined: Jun 12 2022 - 23:35
Posts: 111
softwarejanitor wrote
softwarejanitor wrote:
magnusfalkirk wrote:

MacFly,

Thank you for this and everything else that you have done to help make the Dan][ card even better. Granted Dan built this usefulness into the card from the beginning but that I know of you're the first person to do something to make it easy to use and easy to update.

 

Yeah thanks for McFly for picking this up and running with it.  I've been working on another project very intensively and haven't had too much time to dedicate to the Dan II controller.  I've been working on an emergency radio ( http://www.github.com/profdc9/RFBitBanger ) and I hope to have a kit released before August.

Dan

Log in or register to post comments