Discussion:
[pyusb-users] a dream of uniform serial-usb handling...
Chapman Flack
2015-04-19 00:53:17 UTC
Permalink
Hi, I'm new to pyusb, thinking I have an idea to propose for how
uniformly serial and usb support in Python could be handled ... it
started with this sort of dream vision:

1. I have this gizmo that once upon a time had an old-fashioned 232
serial interface to the computer. I can talk to it in Python:

import serial
giz = serial.serial_for_url(sys.argv[1]) # caller passes /dev/ttyS0

2. But that was then. These days it's the same gizmo with a
USB-to-serial chip glued on the front. I'd like that to change my
code as little as possible - maybe even not at all except for what
the caller sends as argv[1].

Under the hood a few different things could be going on:

2a. My kernel has a driver for that USB-to-serial chip, and makes a
native device node.

import serial
giz = serial.serial_for_url(sys.argv[1])
# caller passes /dev/serial/by-id/usb-0683_1550-if01-port0

2b. Maybe the kernel doesn't have a specific driver for that chip, but
the chip has the right communications USB class and the kernel's
usbserial generic driver might work with it. Same native device node
gets created, so nothing changes in Python code, not even the device
name.

2c. Instead of using an in-kernel driver, maybe pyusb is around and the
chip is from FTDI...
import serial
import ftdi.serialext # ?? see more dream below
giz = serial.serial_for_url(sys.argv[1])
# caller passes ftdi://0x0683:0x1550/2

2d. Maybe the chip's an MCT instead? (caution, science fiction ahead)
import serial
giz = serial.serial_for_url(sys.argv[1])
# caller passes mct://0x0711:0x0210/1

2e. Could there be a generic communications-USB-class driver, like the
one the linux kernel has? (Also science fiction at the moment.)
import serial
giz = serial.serial_for_url(sys.argv[1])
# caller passes usb://0x0683:0x1550/2

Ok, waking up from the dream ....

The current state of things is almost where it could allow all the above
to work.

- pyserial already provides serial_for_url() that can open a native
serial device node or a "url" with extensible schemes via
protocol_handler_packages. It already provides a SerialBase class and
several implementations.

- There's already a UsbSerial base class that extends SerialBase and is
used as a mixin for the pyftdi driver allowing it to be used as a
pyserial driver. This should work for other USB-to-serial drivers
(prolific, mct, whatever) too, or a generic communications-USB-class
driver.

- That UsbSerial class, and the rest of the 'serialext' package, was
contributed to pyftdi by Emmanuel Blot, but maybe pyftdi isn't the
best place for it, because it could be a base mixin for many different
USB-serial driver protocols and ftdi is just one specific one.

- Maybe it would make sense if pyftdi.serialext became pyusb.serialext
instead? It is BSD-licensed (unlike the rest of the code in pyftdi,
which carries LGPL headers) so that much should be a comfortable fit
with pyusb.

- That way, any Python driver for a USB-serial chip would probably
import pyusb (as it likely would anyway) and extend UsbSerial.

- Moving serialext would require Emmanuel Blot's agreement for pyftdi,
and for a while pyftdi would probably have to do try: import
pyusb.serialext; except ImportError: import pyftdi.serialext; to work
in old and new cases.

- UsbSerial does import some UsbTools code from pyftdi proper (LGPL),
most significantly parse_url, which is useful. It would be good to
have one url-parsing scheme applicable across any USB-serial drivers,
not just as part of pyftdi. Maybe Emmanuel could be approached about
allowing BSD licensing for usbtools.py?

- At the moment there isn't a generic,
don't-care-what-the-chip-is-as-long-as-I-can-bulk-read-write-it
pure-pyusb driver like the usbserial generic one in the linux kernel.
I might be interested enough to try writing one at some point. Maybe
such a generic usb-serial driver would also be at home within pyusb?
I realize both of these proposals turn pyusb into something slightly
more than a ctypes wrapper on libusb. Maybe that's not desired, and
there should just be a package usbserial in its own right, with
relationships like

pyserial pyusb
| |
+--------------+-------------+
|
serialusb
(UsbSerial base class,
generic USB-class-based driver)
|
+-------+-------+------------------...
| |
pyftdi other specific USB-serial chip drivers...


On the other hand, if there is no objection to growing pyusb a little
bit beyond wrapping libusb, that would save creating another separate
package.

Reactions? Does my dream make sense to other pyusb-users readers?

Thanks,
Chapman Flack
Emmanuel Blot
2015-04-19 11:31:14 UTC
Permalink
Post by Chapman Flack
Reactions? Does my dream make sense to other pyusb-users readers?
Hi,

A quick note about the licensing scheme of pyftdi (and friends).
Some parts are LGPL-licensed because I've initially developed cftdi on
top of libftdi - a Python-to-C wrapper for libftdi, and used an API
which is very close to libftdi, which is released as LGPL.

At that time, I've been in touch with one of the authors of libftdi to
ask whether I could use the same API (and some part of the code,
although ported from C to Python) with a MIT-style license. To be
honnest, LGPL is quite hard to master, LGPL with Python is - for me -
nearly impossible to understand. For instance, what is "static
linking" or "dynamic linking" in Python's world?

Anyway, LGPL in this case was so fuzzy to interpret, and because I had
no time to dig into the licensing "mess" I chose to keep the same
license as libftdi.
From my point of view, I used to be in favor of GPL-like licenses as a
hobbyist, but as a profesionnel developer, this license scheme keeps
giving me headaches, even after some dedicated training...

I always release new code under MIT / BSD 3 clauses. I wish I could
release pyftdi under MIT license as well, if somebody who master
licensing issues can tell whether it could collide or not with the
distant roots of pyftdi, that is libftdi.

To sum up: I have no objection to move to MIT license the whole pyftdi
package, I just do no want to deal with this licensing nightmare. Any
advice on this topic is very welcome.

Cheers,
Manu
Wander Lairson Costa
2015-04-21 13:17:53 UTC
Permalink
Hi,

I am not very versed into pyserial and ftdi stuff. One question comes
to my mind is: what's the advantage of moving stuff from pyserial to
pyusb?
I have an idea of providing an usb.classes package, which would ship
modules for each USB standard class. But I didn't elaborate onto this
yet, it is currently only a vague idea. This might be what you want
here.
Post by Emmanuel Blot
Post by Chapman Flack
Reactions? Does my dream make sense to other pyusb-users readers?
Hi,
A quick note about the licensing scheme of pyftdi (and friends).
Some parts are LGPL-licensed because I've initially developed cftdi on
top of libftdi - a Python-to-C wrapper for libftdi, and used an API
which is very close to libftdi, which is released as LGPL.
At that time, I've been in touch with one of the authors of libftdi to
ask whether I could use the same API (and some part of the code,
although ported from C to Python) with a MIT-style license. To be
honnest, LGPL is quite hard to master, LGPL with Python is - for me -
nearly impossible to understand. For instance, what is "static
linking" or "dynamic linking" in Python's world?
Anyway, LGPL in this case was so fuzzy to interpret, and because I had
no time to dig into the licensing "mess" I chose to keep the same
license as libftdi.
From my point of view, I used to be in favor of GPL-like licenses as a
hobbyist, but as a profesionnel developer, this license scheme keeps
giving me headaches, even after some dedicated training...
I always release new code under MIT / BSD 3 clauses. I wish I could
release pyftdi under MIT license as well, if somebody who master
licensing issues can tell whether it could collide or not with the
distant roots of pyftdi, that is libftdi.
To sum up: I have no objection to move to MIT license the whole pyftdi
package, I just do no want to deal with this licensing nightmare. Any
advice on this topic is very welcome.
Cheers,
Manu
------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
pyusb-users mailing list
https://lists.sourceforge.net/lists/listinfo/pyusb-users
--
Best Regards,
Wander Lairson Costa
Chapman Flack
2015-04-26 19:50:38 UTC
Permalink
Post by Emmanuel Blot
A quick note about the licensing scheme of pyftdi (and friends).
Some parts are LGPL-licensed because I've initially developed cftdi on
top of libftdi - a Python-to-C wrapper for libftdi, and used an API
which is very close to libftdi, which is released as LGPL.
...
I have no objection to move to MIT license the whole pyftdi
package, I just do no want to deal with this licensing nightmare.
I understand. I think what I was proposing was less drastic: I think
the UsbSerial class (and indeed the whole serialext package) is more
general than just for FTDI chips, and would make more sense as a part
of pyusb. Then all different USB-serial chip drivers, including pyftdi,
could share it as common code.

In my first message I wondered whether pyusb wanted to grow beyond just
Post by Emmanuel Blot
Until 0.4 version, PyUSB used to be a thin wrapper over libusb.
With 1.0 version, things changed considerably. Now PyUSB is an
API rich, backend neutral Python USB module easy to use.
That convinces me even more that for pyusb to absorb the serialext
common code and UsbSerial base class from pyftdi would make sense
all around.

I think everything in serialext already *is* BSD licensed, so that
much should be easy. The only sticking point is that it makes some
incidental use of UsbTools from the LGPL part of pyftdi. The most
significant thing used there is parse_url. Did you also derive
parse_url from something in libftdi? I just looked over the libftdi
API and I didn't see something similar.

If parse_url was something you added, then maybe it is not a
problem for you to BSD license it to match the serialext code,
and it could also be raised to the status of something more
general in pyusb, where it could unify the way you specify a
device whether pyftdi or something else is the driver.

In fact, if we could do that, it might be worth thinking out
a url scheme that applies even to non-serial USB devices. I was
just setting something up recently to talk to a device that needs
only control transfers, not a whole serial interface, and it would
be nice to be able to specify that device as a URL also.

I'm only beginning to think about what a general usb url scheme
might look like, so take this as brainstorming, but maybe the
'scheme' portion should just be 'usb://' and I could pass something
like usb://0x067b:0x2303 as an argument to a script that just does
a pyusb device lookup and a control transfer. Or if I have a script
that expects some kind of serial port and uses pyserial. I could
pass it /dev/ttyS0 or /dev/ttyUSB0 if the kernel driver is in place,
or usb://0x0683:0x1550?driver=ftdi and it gets a Serial instance in
every case and Just Works.

If the url scheme gets generalized a bit away from what's now in
pyftdi, maybe that also moots the license question around the existing
parse_url. But perhaps it is still good to BSD license the existing
parse_url if that is possible, just so there is less to worry about.

What do you think?
-Chap
Emmanuel Blot
2015-04-28 08:19:40 UTC
Permalink
Foreword: Please do not reply to digest messages (or without editing
the subject), as tracking threads is quite difficult :-) Thanks.
Post by Chapman Flack
I understand. I think what I was proposing was less drastic: I think
the UsbSerial class (and indeed the whole serialext package) is more
general than just for FTDI chips, and would make more sense as a part
of pyusb. Then all different USB-serial chip drivers, including pyftdi,
could share it as common code.
I do not mind if some pyftdi code move to pyusb, but this is really up
to Wander.
I'm not sure it would really make sense though.

I'm sure nobody wants to move the UsbTools class: this code is crap (I
can write this as I wrote it :-) ).
The features it provides are definitely useful, as enumerating a full
USB bus topology takes so much time that it should be done as few
times as possible. However the way it is implemented is really ugly
(it was a quick and dirty hack that has not been removed) and
rewriting it is my todo list for years...
To sum up: rewrite it, do not import it if you really need it.

The only potential issue with licenses is the ftdi.py file. All other
ones do no need to use the LGPL v2 license.

Manu.
Chapman Flack
2015-05-02 01:38:11 UTC
Permalink
Post by Wander Lairson Costa
I am not very versed into pyserial and ftdi stuff. One question comes
to my mind is: what's the advantage of moving stuff from pyserial to
pyusb?
I might have needed to give a broader view first. I did not intend to
move anything from /pyserial/ to pyusb. Those two have a sort of equal,
independent relationship.

The use of pyserial is for any Python code that wants to use a serial
comm port. The code should be able to pass some provided string to
serial.serial_for_url() and get a Serial instance back, and use it.
Whether the actual port is a UART on the motherboard, or some tunneled-
over-IP thing, or a USB-to-serial chip with a driver in the kernel, or
a USB-to-serial chip with a driver built with pyusb. So pyserial is a
common API for serial ports whether they are USB or not.

Respectively, pyusb is a common API for USB devices whether they are
serial ports or not. :)

The only interdependency is that if somebody uses pyusb to write a driver
for a USB-to-serial chip, it ought to be written to supply a class that
derives from serial.Serial. And somehow, that driver needs to get called
when somebody passes a certain form of string to serial.serial_for_url().

Those two pieces of glue right now are in the package pyftdi.serialext.
But FTDI is only one brand of (perhaps not fully standard-class-compliant)
USB-to-serial chip, and package pyftdi is really a hardware-specific driver
for that chip ... all except for the subpackage pyftdi.serialext which is
/not/ specific to that chip. It's just the same glue code that any
pyusb-based USB-to-serial driver would have to duplicate in some form.
It's just that Emmanuel wrote it first, and put it in pyftdi. :)

It defines the useful base class UsbSerial that is both an instance
of serial.Serial and proxies a pyusb device. It also contains the code
to act as a protocol-scheme hook into serial.serial_for_url and
recognize strings that should be USB devices.

These are the pieces of code that I think could be hoisted up from
pyftdi and would reasonably be in pyusb for general use by pyftdi
and any other similar drivers for USB-to-serial chips from MCT or Prolific
or Cypress or whomever.
Post by Wander Lairson Costa
I have an idea of providing an usb.classes package, which would ship
modules for each USB standard class.
I like that idea. And if that existed, the package for class 02 would
reasonably be the right place for this common code.

I'm sure you've noticed how usb.org defines standard class protocols
and there are some devices that actually work that way, but others rush
to market and provide similar functionality but not with the standard-
defined protocol. For example (the first one I crashed into), the Audio
class defines a USB MIDI protocol, but manufacturers like Midiman make
USB-MIDI devices that work with their own strange protocol. Having a
driver for the standard class protocol is great because it will work with
any device that complies, but as soon as you attach one of the strange
devices you need its own strange driver code.

The same situation seems to exist for serial. There is a communications
class spec that defines a standard protocol, and if there were a pyusb
driver for that protocol, it would work with any compliant device (for
example, Cypress). I think it would be good if such a driver could be
developed, and even distributed as part of pyusb in the usb.classes package.

That would not end the need for strange nonstandard protocol drivers like
pyftdi, as long as strange chips like FTDI are being sold. I can imagine
a possible rule that only drivers for strict USB class specified protocols
might be part of pyusb in usb.classes, and drivers for non-class-compliant
hardware would remain in separate packages like pyftdi.

But it would make sense that usb.classes.communication (or whatever it
should be named) contains the common glue such as the proxy class derived
from Serial and the hook to intercept serial_for_url. The ideal would be
that if somebody had Python code that wanted a serial port, they could
pass argv[1] to serial_for_url() and get a usable Serial instance back,
whether the argument was /dev/ttyS0 or /dev/ttyUSB3 (in-kernel driver)
or something like usb://0x0683,0x1550/?drv=ftdi or usb://0x04b4,0x0003/
(which would be a Cypress chip and Just Work with the class standard
driver because Cypress implements the standard)....

Does that clarify the structure I had in mind?

-Chap

Loading...