UbuStream Dynamic Source "mini-plugin" architecture for on-demand streams

[ubu]

The next release of UbuStream (v2.0) will include support for "mini-plugins" that will scrape content from web sites that publish on-demand streams (often updated daily). This will allow UbuStream users to watch the most recent streams without having to manually set them up as individual "Stations". I'm starting this thread so potential users (and authors) can discuss issues relating to the architecture for these "Dynamic Source" apps.

Overview

At startup, UbuStream will retrieve a list of all Dynamic Source applications in the gbpvr\Plugins\UbuStream\DynSource folder and, if not already present, will add them to the UbuStream database.

When the user views each Dynamic Source (using the UbuStream config app) the DynSource app will be asked to publish a list of "sections" available on its target web site. The user can then select the sections from which they want to receive automatically updated content and, for each section selected, UbuStream will create a "Station Group" (this is a new feature in UbuStream v2.0).

At run time (when selected from the GB-PVR menu), UbuStream will call the DynSource app, read the playlist file it produces and populate the Station Groups with the new items (for those sections they have selected).

Dynamic Source Applications

A Dynamic Source application has two execution modes. If executed with the parameter "-sections" it will produce a file containing the section names available on the target site. Executing it with no parameters will produce a file of all the currently available stream items in all sections.

For a mini-plugin that targets a site that doesn't have "sections", the section name in the output could just be hard-coded by the author and a single "Station Group" would be created with that name.

Provided the app conforms to the naming standards and file formats described below, it can be written in any language and use whatever technique the author chooses. However, to make coding a DynSource app easier and to encourage people to "have a go" at authoring one, a common class library (DynamicSource.dll) will be provided that will take care of most of the heavy lifting (see description below). Using this library, authors can write a simple program in any .Net compliant language (C#, VB, Python, etc.) that contains one method to parse a web page for section names and another method to parse a page for stream URLs.

Naming standards

Dynamic Source name: <name> (eg. CNN)
Mini-plugin app: <name>DynSource.exe (eg. CNNDynSource.exe).
Sections file: <name>Sections.plx (eg. CNNSections.plx)
Playlist file: <name>Playlist.plx (eg. CNNPlaylist.plx).


PLX file formats

Although the Dynamic Source file format is based on the widely accepted standard PLS format, some liberties are taken with the original format, so the files should be given the extension '.plx' (PlayList eXtended).

for playlist file:

[playlist]
File1=<item stream url>
Section1=<name of section containing this item>
Player1=<(optional} name of media player to be used for this item's stream>
Type=[I]<(optional) stream type (Audio or Video)>
Description1=<(optional) item description text>
Website1=<(optional) url of a web page containing schedule info>
Title1=<item name>

for sections file:

[sections]
Player=<optional name of media player to be used for this section's streams>
Section1=<name of section>

Notes:
- "Optional" lines may be omitted.
- The Player1 line in the playlist file allows the default player for the Dynamic Source to be
overridden for that item. If omitted, the default player will be used.
- The Player line in the sections file declares the default player for the Dynamic Source streams
and should only occur once. If omitted, UbuStream will use the player entered in the config panel.
- If the Type line is omitted, UbuStream will default to "Video". This is only required for Audio streams.

DynamicSource.dll class library

The class library provides two methods which may be overridden by any class in the DynSource app which inherits the DynamicSource class:

• Get_Sections()
  
• Get_Section_Items(SiteSection section)

and some "utility" methods which may be called by the DynSource app (usually you won't need to):

• Read_HTML_Page(String targURL)
  
• Publish_Section_Names(String dynSourceName, String dynSourcePlayer)
  
• Publish_Site_Items(String dynSourceName)
  
• Write_Sections_File(ArrayList sectionList, String dynSourcePlayer, String plxFile)
  
• Write_Playlist_File(ArrayList itemList, String plxFile)
  
• Get_Gbpvr_DynSource_Folder_Name()

I will be providing a "beta" version of the dll, together with some sample DynSource apps, very soon. Watch this thread.

[HTPCGB]

Attached is the updated CNN mini-plugin.

[ubu]

Attached is the updated CNN mini-plugin.

Excellent! Works fine. Three very minor points:

1) The first line of the Sections file should be [sections] not [playlist]. It will work OK in either case currently but I'm reserving the right to require the corect header in the future. (And it would be good if the output of your "mini-plugin" and the output of mine have identical formats).

2) If you add the line Player=Windows Media Player as the second line of the Sections file, it will ensure that the correct media player will be used for playing the streams associated with your DynSource. It'll work OK without it, but the user will then have to remember to manually select the correct player for the CNN DynSource in the UbuStream config app before using your app for the first time.

3) Your app is currently placing the output files in the current directory from which it is executed (which I agree is normal behaviour for this kind of app). When run from a GB-PVR plugin, this puts them in the gbpvr directory and, in an attempt to keep from cluttering up that directory, I've been putting my plx files in the gbpvr\Plugins\UbuStream\DynSource directory. I just do something like:

Code:

String playlistFile = myDynSourceApp.Replace("DynSource.exe", "Playlist.plx");
String sectionsFile = myDynSourceApp.Replace("DynSource.exe", "Sections.plx");

inside my mini-plugin app. This ensures that the output files always end up in the same directory as the app itself. If you prefer to stick with using the current directory, I can change UbuStream (and my mini-plugin dll) to do that instead. It's just a little simpler for me to keep everything in the DynSource folder though. It's your call. Let me know which you prefer.

Thanks for the rapid turnaround. I'm very close to completing this now.

[stefan]

Cool... I managed to add like 40 or so "SR" (Swedish national radio) to the worldwidemediaproject: http://www.worldwidemediaproject.com/rad...&genre=all

But realize that the URI might change over time. Would be cool to write a mini-plugin for them, but I wouldn't know where to start. I'm a java programmer, so I don't have a clue how to write c#... perhaps not too hard to learn at least the basics, but it'd be a lot easier if I had a starting point Is there any example source code I could start with?

[ubu]

Cool... I managed to add like 40 or so "SR" (Swedish national radio) to the worldwidemediaproject: http://www.worldwidemediaproject.com/rad...&genre=all

But realize that the URI might change over time. Would be cool to write a mini-plugin for them, but I wouldn't know where to start. I'm a java programmer, so I don't have a clue how to write c#... perhaps not too hard to learn at least the basics, but it'd be a lot easier if I had a starting point Is there any example source code I could start with?

I'm just about finished with the class library and sample mini-plugin apps (see my first post in ths thread) so I'll be "publishing" it here. If you already write java you should have no problem modifying one of my sample apps to your needs. C# isn't wildly different from java.

You just need to write two simple methods. The class library methods will load HTML into a string for you, so you'll just need to parse the string to find your URIs and stick them into a collection. (I use regular expressions to do this, because I'm comfortable with using them, but any string-slicing technique is fine). The class library will take care of writing the collection to the output file. In fact you don't even need to use C#. The class library should work with any .Net compatible language as long as it allows you to inherit the main DynamicSource class and override a couple of methods.

[stefan]

Looking forward to it, but not promising anything. But it would be fun

[ubu]

I'll give it a try for Uitzendinggemist (a Dutch site which streams shows that have been on TV last week)

Personally, i would prefer exchanging data via classes in dll's instead of via .pls/.plx-files. Currently i've got a prototype up-and-running of a webvideo-plugin that uses dll's for specific website-parsers.

That would be great. HTPCGB has one working for CNN, I've got 3 going (BBC, Mosaic and DemocracyNow) so, if you add one for Uitzendinggemist, maybe it's time for me to start thinking about a wiki sub-page for Dynamic Source mini-plugins.

I thought about using classes instead of files for the data interchange but:

1) I want the mini-plugin apps to be both easy to write and not too dependant on any particular programming environment. Right now, you can write a Dynamic Source app in any language (Lisp, REXX - Fortran even) and, as long as the app can write its output in the plx format, it will work fine within the Dynamic Source architecture. Using a class to transfer the data would have constrained the apps to be written in .Net languages. (However, for those people who are using .Net languages, I'm providing the DynamicSource.dll class library so they don't have to reinvent the wheel.)

2) UbuStream lets users invoke the app both in real-time and background mode. When they click an "AutoSynch" button the mini-plugin app is invoked to scrape the data and, when it completes, the data is imported. For this, using classes instead of files would have worked OK. But there is also an option to run the apps in background mode, with no user interaction required. Each time UbuStream starts up, the app is fired off in a separate process, allowing UbuStream to continue without any noticable delay. A FileSystemWatcher object monitors the plx files and, when one changes, triggers the data import method(s) . I imagine you could arrange to do something similar using classes instead of files but I suspect it would involve complex thread management code (way above my pay grade - or maybe I'm just too lazy ).

Is there some way we could integrate your existing work into this architecture? A wrapper that would dump your dll data into a file, perhaps?

[HTPCGB]

Sorry for the silly mistakes, that's what I get for coding while watching TV.

[ubu]

Sorry for the silly mistakes, that's what I get for coding while watching TV.

The file format is perfect now. However, the files are still getting created in the current directory (the folder it's being executed from) and not in the directory where the app resides (see item 3 in my earlier post). Is that what you intended?

[HTPCGB]

Edit: I'm pretty sure that this should work now.