EPG Design
From The Neuros Technology Wiki
 |
This is a high-level analysis of the EPG functionality for the OSD.
You can edit this, but please correct only minor problems, don't drastically change anything. There is a thread in the mailing list where to discuss this topic, so please discuss away there and I'll merge here the changes that we all agree upon.
Contents |
[edit] Constraints
EPG data can take a lot of space. From my experiments i have a 600K database for 185 channels and 2 days of simple data (programme name, start time, end time). And we want data that is a lot richer than that (episode titles, synopsys, cast actors, etc), and for at least a week.
Even if there was space processing and converting that large amount of data from whatever format the provider uses (XML, proprietary DB, text files) it would strain the OSD resources too much.
So I've come up with this solution that involves an intermediate conversion step on a "server" machine (see below).
[edit] Basic idea
The most likely idea is that Neuros gets an agreement with one or more EPG data providers (tvtv, zap2it or titantv, and/or others).
It would also be nice to use data from IMDB or similar to flesh out movie listings. ~~vzl
Then we have a script that runs on a cron job on the Neuros servers and fetches data periodically from these providers, digests it and stores it in a central database.
Then the OSD units will run something like the firmware auto-update periodically, and get data out of this central database and update their on-board local EPG databases.
Now, i'm not your typical TV couch potato, but I suppose that people generally watch a dozen channels, maybe 50 at most. So we ask the user to setup their OSD with a list of their favorite channels (with a limit) and then the EPG auto-update downloads only data for these channels, not for everything.
Then we have the EPG application proper. More on this below, but here's a small recap of what it does: It allows the user to browse the EPG data using various types of views, view all the details we have about programmes. It allows searching for specific programmes, or other searches we might want to implement. It allows to change the favorite channels list (which in turns triggers the auto-update again for the new channels) and also setup channel numbers for the various stations (used to change channel with IR blaster). And of course it allows to view/remove the scheduled recordings.
Once the user schedules some recordings, they are saved in a local database. NMSD daemon will look into that database and when the time comes it will use the IR blaster to change to the channel specified in the recording and then start recording.
Here's a more detailed analysis of the various pieces.
[edit] Neuros server side
[edit] Gathering data
This is pretty simple, we just need a script on cron that picks up the data from the providers, whatever way they choose to provide it (XML-RPC, SOAP, plain file download). Then the script parses the data from whatever format it is in and stores it into a central database (i'm thinking a classic mySQL DB, we already have one i think).
[edit] Serving data
The server needs to have a mechanism to serve up data only for a subset of channels. We can do that in two ways.
The simplest route is that we store a lot of small update files on the server, one for each channel, in a public accessible HTTP area, then we pick them up from OSD and merge them with the OSD on-board DB.
The other route is that we have a web application running on the neuros server that receive requests and serves up dynamically a DB file tailored for the channels that the user needs, for the time span that the user needs. Then the user can simply save that single file on the OSD immediately, replacing the old one they had. Or optionally serve small incremental updates only for the requested channels that the OSD will merge like the in other option.
We can start with the first option and add the second later when there's time to implement it as a web application.
[edit] OSD side
[edit] Database
[edit] EPG Database
All access to this database MUST be done using the #Data Model
This will be an SQLite3 database. I picked this format since it's small, serverless, and database files are self-contained. It's also reasonably fast and allows to use most of SQL syntax, which comes in handy for the searching functionality.
More importantly, this format is also used by XMMS2 already, so we just re-use the same library and we save on ever so precious flash space. Other suggestions will be considered, but you need to be very very convincing ;)
This will have tables that store the following things:
- full list of all available channel names
- favorite channels for the user (only those will be updated/viewed/processed by the OSD)
- channel lineup (that is, association between channel names and numbers (needed for IR blaster))
- programme data (airtimes, basic and extended information on all programmes) (this depends on what the various providers can give us, but at least it has programme title, episode title, start time)
Precise database layout will be provided later, but not relevant for this stage of discussion.
[edit] Scheduled Recordings Database
All access to this database MUST be done using the #Data Model
I prefer to have this too inside an SQLite database. I'm not sure where this is stored right now, but let's move it to SQLite. This gives us more flexibility and allows us to link with the #EPG Database which hold the channel linuep information.
In this database for there are table to hold the following data:
- The current scheduled recordings. For each scheduled recording we store:
- Start time (note that this can be different than programme start time, if the user overrides it)
- End time (same as above)
- A link to the programme in the EPG Database
- A link to the channel in the EPG Database
- Channel number override (NULL if no override)
- TODO: Type of remote from IR Blaster Database override (NULL if no override).
- the recordings that were already completed
More accurate database layout will be provided later.
[edit] Data Model
ALL access to the #EPG Database and #Scheduled Recordings Database need to be done using this API
This is just a small library API that abstracts the access to the database files (so we can change format later in the future and need not to change all applications that use it).
Example of APIs:
- enumerate all favorite channels
- enumerate programmes for a certain channel that happen between 15:00 and 20:00 of May 15th
- add/remove a channel to favorites
- get detailed data for a programme (synopsis, plot, cast, etc)
- schedule a programme for recording
etc. etc.
I will submit a detailed design for this API later. For current discussion it's not necessary.
[edit] EPG Updater Daemon
I really like this to be implemented as a daemon, not part of any application. This allows it to be able to completely self-contained and deactivable for people who don't need it. It will read from the global OSD settings for how often to update and for how much days in advance the user needs data.
When it's time to update, it reads the following things:
- favorite channels (from the EPG database)
- current location (from OSD settings, see below)
- how many days of EPG data the user wants (from OSD settings)
Then sends this stuff down to the neuros server and the server sends back a database update. Once we get that update, the daemon merges it with the on-board EGP Database and then goes back to sleep.
There will also be a simple IPC interface to trigger forced updates (used for example when the user adds a new favorite channel).
This component will also be responsible for updating the channel lineup from the server. This update is triggered via IPC from the Settings application (see below) when the location (Nation and ZIP code) or the broadcaster is changed.
The lineup download is small, since it's only channel name and number, optionally a tiny icon.
TODO: if the network is down, this download shall be scheduled for later on. Or simply a message shall be shown to the user telling they need to be online to be able to set these EPG options.
If the new channel lineup is different from the old one, we need to do the following operations:
- delete all scheduled recordings for channels that do not appear anymore in the new lineup. Since this check can be fairly complex, we can just delete all scheduled recordings on the first implementation.
- delete all favorite channels that do not appear in the new lineup anymore. In a later implementation we can keep an history of favorite channels for each linuep, so that users who move their OSD a lot or use it with different broadcasters don't have to input again their favorites every time.
[edit] EPG GUI Application
[edit] Settings
This will either be a separate settings app, or a screen in the EPG application (depending on how the split of osdmain happens once we have the refactoring done). Anyway it will allow to input the following settings:
- How frequently check for EPG updates ? (default 3 days ?)
- How much days of EPG updates to fetch ? (default 1 week ?)
- Nation where the user is located:
- If USA being able to also select first a ZIPcode and then cable/sat/air tv broadcaster from a list of those who operate in that ZIPcode (for automatic lineup selection). Depending on space constraints all the ZIPcode/provider associations might be already stored on board, or else we need to interact with the server once the user enters the ZIP to fetch the list of all broadcasters for that code.
- For other nations there might be a similar mechanism, or not (depending on the kind of data from providers).
- We should also provide a location named "Custom" for which the user can write their own linuep (see "Channel linuep manual setup" below)
Changing Nation+ZIPCode+Broadcaster will trigger EPG Updater Daemon to download the corresponding channel lineup from the server (see above)
[edit] Channels
[edit] Favorite Channels
This allows people to select a small number of favorite channels among all the channels available on air (more than 600 possibly). Typically no more than 50. We will probably have a pre-defined list of channels on out-of-box OSDs which are the most popular for North America (for better out-of-box user experience).
The initial list of channels (linuep) will be downloaded when the user sets up his zipcode and broadcaster (see Settings).
The selection of favorite channels out of that list can be done in 2 ways:
- by using the current cooler lists, and adding a checkbox/color-toggle functionality to them, thus allowing for multi-selections.
- by implementing a grid widget, and showing channel names with color-toggles for selection. A grid allows for seeing more channel names at once and makes it faster to select stuff for the user (grid widget can also be re-used in other places).
Once the user has selected his favorite channels, a forced epg-update is triggered for the changed channels.
[edit] Manual Setup Of Channel Lineup
This is an optional panel, but it can be necessary for people who have strange channel lineups that the provider doesn't have (or for most european countries, i think). Users don't need go to this panel most of the time, but in the rare cases when they need this is an important panel.
Basically i am designing it as a normal list that shows all the favorite channel names, and beside each name the channel number. When the user is over an item, he can press the numeric keys on the remote, and edit the channel number.
Saving this will save the lineup to the DB, for the "Custom" location (see above)
[edit] Views
I'm mentioning views as plural because there was considerable debate on IRC and it was pretty clear that there were 2 very popular views of the EPG and that neither of them was worth leaving out. By doing it this way we also leave room open if we want to add other views later down the road. One of the two views will be of course the default one the program appears initially. But user can choose the other (either via XIM or make the other default in settings, maybe)
[edit] Column View
This will be much like what TiVo has, with some changes to make it better customized for the OSD.
Basically a left and a right column, plus a bottom panel. Left column is a simple list of all favorite channels, that the user can navigate. Each channel row displays channel name + currently airing programme title (variation: next airing programme and start time)
The right column displays programmes for the channel selected on the left list, starting from the currently airing and going forward (depends on font size and such we can have 8 or more rows of future programmes). Each programme displays starting time + programme name. Optionally programme names that don't fit can have a scrolling animation (like long song titles on the old Neuros2) Scrolling down this list will show further programmes for the future, cycling over to the next day(s). If a programme is already scheduled for recording, its row should be displayed highlighted in some way (or with some icon).
User can change focus to both columns with LEFT/RIGTH keys. When right column is focused, pressing XIM on a programme row brings up the menu. This menu is identical to the XIM menu described in the #Row View
Pressing record will bring up panel to allow user to modify or confirm default recording options.~~vzl
Variation: pressing RIGHT when already on right column will bring up menu to schedule currently selected programme.
Variation: pressing REC will schedule immediately (no menu) the currently selected programme.
Bottom panel shows more detailed info for currently selected programme (episode title, cast, synopsis, etc).
[edit] Rows View
This is kind of like some of the visualization in MythTV. It also looks kind of like this web visualization. Or even better this OSD mock-up.
Basically we have a series of rows, each row represents a timeline for a single channel. At the top of the grid there's an header which looks like a rule with a notch each half-hour. At most three hours at a time are displayed. This is the time span each row covers. This header also displays the day that we're currently browsing.
First column of each row is the row header and stays fixed. It displays the channel name. (variation: channel icon when available)
On each row we display the programmes horizontally, with boxes as large as the time they cover. Each block can either display only the programme title (we have more rows that way) or it can display also the episode title (or movie information).
We can also color the block a different color according to the "genre" or the programme (movie, tv show, sports etc). The block should also be displayed highlighted in some way (or with some icon) if that programme is already scheduled for recording. Programmes that have a duration longer than what can be displayed on screen need to have a symbol that tells the user that it continues on the next block (for example a little arrow to the side).
Pressing UP and DOWN will move between channels (= rows), scrolling to the top like a circular list when reaching the bottom. Pressing LEFT and RIGHT will move between programmes for that channel (= blocks on the same row). The currently selected programme should be highlighted in some visible way. When scrolling over to the left or right outside of the 3 hours that can be displayed on-screen, then all the visible rows should scroll in that direction by increments of 1 hours (2 notches on the ruler header). The ruler header shall be updated accordingly.
When a programme is selected, and not scheduled, the XIM menu will have the following options:
- Schedule this program for recording. This will write into the #Scheduled Recordings Database a new schedule for the current programme and channel, with same start/end time and with no overrides (see also next item).
- Custom Schedule for this program. This will do the same as the above, but it will also bring the user immediately to the #Scheduled Recording Editor to edit various recording options if the defaults are not ok.
If the programme is already scheduled, it will have instead the following options:
- Delete schedule
- Go to #Scheduled Recording Editor to change recording options.
Variation: pressing REC will schedule immediately (no menu) the currently selected programme.
We can also have less rows on screen but display an area at bottom where we display extra information for the currently selected programme (similar to what we have on the Columns view, see above).
[edit] Recordings (Scheduled and Completed)
[edit] Scheduled Recordings List
This is a simple list where each item is a scheduled recording. For each item it shows the date/time when it will start, and the name of the programme (with episode number and title, if it's a tv show). The list is sorted by time, with the recordings that will happed soon at the first places. Other sorting options are possible (selectable via XIM) but not necessary in the first version.
If would be nice (but not necessary in the first version) to show the date/time in a more "human-friendly" way. For example if the recording it today, just show the time. If the recording is this week, just show the week day name (e.g. "Wed 19:00").
When an user stays over a single item, a popup will appear that shows more detailed information about the programme (like what happens in the file manager).
The XIM menu will have the following items:
- Delete the recording
- Open the #Row View or the #Column View and select the programme.
- Go to editing the recording details in #Scheduled Recording Editor
[edit] Scheduled Recording Editor
This panel will display some details of the scheduled recording and at the same time allow to change options. Normally the user have no need to access this panel, since the defaults for scheduled recordings should be already ok. However there might be times where this is needed.
This will just display the following information: Programme title, episode number/title (if any), short description, duration of recording (end time - start time, synchronized to what the user changes below).
It will allow to edit the following information:
- Start time
- End time
- X minutes to record before and after start/stop times (convenient to pad recording in case broadcast isn't exactly on time)~~vzl
- Location where to save the recording
- Channel number (this is just an override of the number, the channel itself remains the same)
- TODO: Type of remote used by the IR blasting (this will be added when the IR Blaster Database is available).
When saved this will update the #Scheduled Recordings Database if there was any change.
[edit] NMSD
I think that NMSD should be responsible for executing the scheduled recordings.
NMSD needs an internal routine that we call "schedule refresh" that does this:
- It deletes from the #Scheduled Recordings Database all schedules that are in the past (they are useless waste of space)
- It reads in memory the list of all the valid scheduled recordings.
- It sets up a timer to the time of the first scheduled recording.
When NMSD starts up it does a "schedule refresh".
When the timer expires this happens:
- we read the channel lineup table for the scheduled channel and find the channel number (or we use the channel number override if present for the current recording).
- TODO: if there's an IR Blaster override for the current recording, we fetch from the IR Blaster Database the codes we need to change channel. Otherwise we use the default ones.
- we use the IR blaster to change channel.
- the recording starts.
When recording ends or is completely stopped we do a "schedule refresh"
NSMD should also have a command that other apps can call that forces a "schedule refresh". This will be called by applications when they update the scheduled recording database from the outside.
[edit] User's PC side
This is simply an option for people who want to run their own EPG data fetchers. They will be provided by 3rd party devs, like sorune or NDBM sync manager were provided for the old Neuros 2. These will mostly be targeted at nations not covered by official neuros server updates. They will probably use xmltv screen scraping techniques. We don't really care.
These customized fetcher will output a self-contained database package like the one the main neuros server providers (there will be a specification for this that 3rd party developers will need to follow). The users can then update their OSD by putting it on a media card (or some other way), and running some "update from media card" function.
This User-side stuff is low priority, I just added it in for completeness.
[edit] What is missing ?
This is stuff that is useful but only when the rest of the system is up, so we can just fill the details on this later.
[edit] Resolution of conflicting recordings
This should either be:
- automated: the GUI does now allow people to schedule 2 overlapping recordings. This is easier but then the API should be designed to not allow conflicts, not the applications that use the API.
- no-resolution: this means that NSMD will record only one of the conflicting programmes (the first scheduled), and the others will not be recorded.
- manual: there is a panel where all conflicts are displayed and the user must delete one of the two overlapping items. if conflicts are not resolved then NMSD will act as in the no-resolution case above.
- manual (alternative): When user schedules a recording which conflicts with existing recording, immediately a dialog asks which recording the user would like to keep. This is more in the face of the user than the other manual method which requires the user to navigate to a separate panel and resolve all conflicts.
