| Monday, June 20th, 2011, 12:22 UTC | ||
| [12:22:09] | danielk22: | Beirdo: QThreadStorage for the connections created within that thread + thread_deregister().. |
| [12:28:07] | danielk22: | You really need to make sure that connections are only used in the thread that creates them as well though. Otherwise when a thread shuts down the connection may be in use in another thread. So that means the keep-alive code won't work (many of our threads don't have an event loop and keeping the connections alive requires using them...) |
| [12:28:21] | danielk22: | It may be better just to put up with the 5 second delay. |
| [12:29:10] | stuarta: | my preference would be to actually fix the issue |
| [12:32:08] | stuarta: | not that i can talk |
| [12:37:48] | danielk22: | stuarta: fixing is always nice but if fixing it by satisfying the requirements of the Qt4 API would entail more work than the thread port, maybe 160 hours.. i think it could be better spent on other things. |
| [12:38:14] | stuarta: | heh |
| [12:39:16] | danielk22: | but then.. it may be easiest to just fix the Qt code and send them a patch :) |
| [12:40:44] | stuarta: | wonder what that'll break |
| [12:56:50] | Jordack (Jordack!~jordack@h69-131-44-221.mdsnwi.tisp.static.tds.net) has joined #mythtv | |
| [13:25:41] | zombor (zombor!~zombor_@kohana/developer/zombor) has joined #mythtv | |
| [14:22:23] | martin__ (martin__!~quassel@static-88.131.29.2.addr.tdcsong.se) has quit (Remote host closed the connection) | |
| [14:27:49] | kth1 (kth1!~kth@dyndsl-085-016-237-017.ewe-ip-backbone.de) has joined #mythtv | |
| [14:31:27] | kth (kth!~kth@unaffiliated/kth) has quit (Ping timeout: 276 seconds) | |
| [15:03:21] | abqjp (abqjp!~abqjp@97-119-171-42.albq.qwest.net) has joined #mythtv | |
| [15:09:12] | natanojl (natanojl!~jonatan@c83-252-237-63.bredband.comhem.se) has joined #mythtv | |
| [15:12:30] | jmartens (jmartens!~jmartens@109.232.42.33) has quit (Quit: Leaving.) | |
| [15:13:12] | jmartens (jmartens!~jmartens@109.232.42.33) has joined #mythtv | |
| [15:17:51] | stuartm: | davide_: perfect timing, I'd just finished building with the patch applied |
| [15:18:07] | jmartens (jmartens!~jmartens@109.232.42.33) has quit (Ping timeout: 260 seconds) | |
| [15:23:04] | davide_: | stuartm: :) |
| [15:54:12] | jmartens (jmartens!~jmartens@s5597ca60.adsl.wanadoo.nl) has joined #mythtv | |
| [15:56:17] | stuartm: | davide_: all looks good so far, everything that could be recorded in HD will be recorded in HD with the higher priority SD shows moved to another tuner |
| [16:03:55] | davide_: | stuartm: great, that's what it's supposed to do. |
| [16:08:27] | natanojl (natanojl!~jonatan@c83-252-237-63.bredband.comhem.se) has quit (Ping timeout: 255 seconds) | |
| [16:14:56] | j-rod|afk is now known as j-rod | |
| [16:29:58] | RobertS (RobertS!~quassel@20.133.40.85) has joined #mythtv | |
| [16:33:25] | sphery: | danielk22 / Beirdo / stuarta : FWIW, based on the description of Qt-MySQL behavior at http://code.mythtv.org/trac/ticket/7836 , it sounds like the 5s shutdown delay + "Error in my_thread_global_end(): 1 threads didn't exit" is due to some part of our MythTV application trying to use a connection after we've already closed the connections in the connection pool (meaning we've gone down to 0 connections). |
| [16:33:58] | sphery: | so this would be just another place where our code is having problems because we don't have a good way to let the different parts know that we are shutting down |
| [16:34:56] | sphery: | (and the problem is actually that we're doing something after starting shutdown that we shouldn't do--similar to how we were (are?) continuing to process MythProto events and use the context after we've started deleting it and ...) |
| [16:44:29] | davide_ (davide_!~david@mythtv/developer/gigem) has quit (Remote host closed the connection) | |
| [16:44:55] | davide_ (davide_!~david@host103.16.intrusion.com) has joined #mythtv | |
| [16:44:55] | davide_ (davide_!~david@host103.16.intrusion.com) has quit (Changing host) | |
| [16:44:55] | davide_ (davide_!~david@mythtv/developer/gigem) has joined #mythtv | |
| [17:14:50] | Beirdo: | sphery: I have figured out how to fix it (AFACT) |
| [17:26:58] | sphery: | ah, so you're talking about closing the db connection on shutdown from the thread that opened the connection (versus closing connections after each usage of a query) |
| [17:27:07] | Beirdo: | yes |
| [17:27:34] | Beirdo: | basically, the connection is only to be used within one thread (from .open to .close) |
| [17:27:48] | Beirdo: | or rather from .addDatabase to .removeDatabase |
| [17:28:05] | xris: | so more like a web app. :) |
| [17:28:10] | sphery: | including the exec and value and stuff? if so, then we'll need a separate connection pool for each thread, right? |
| [17:28:14] | Beirdo: | if you add in one and remove in another thread, the tracking in QMSqlDatabase gets bent |
| [17:28:27] | Beirdo: | sphery: technically, it seems so |
| [17:28:29] | sphery: | or we'll need our connection pool to be a per-thread connection pool |
| [17:28:36] | xris: | Beirdo: that sounds odd |
| [17:28:45] | xris: | doesn't the threading stuff clean up after itself? |
| [17:28:55] | Beirdo: | http://doc.qt.nokia.com/latest/threads-module . . . e-sql-module |
| [17:29:04] | Beirdo: | it does, but they aren't doing it right in Qt |
| [17:29:15] | xris: | the stuff I wrote for python for work keeps track of each connection in a per-thread local variable. |
| [17:29:34] | Beirdo: | it tries to close it, but the call to the mysql API has to be per thread (it's the way it's designed) |
| [17:29:39] | xris: | the "pool" is global, but it only ever hands out one db handle per thread |
| [17:29:41] | sphery: | I'm wondering if they're just meaning we can't exec() a QSqlQuery in one thread, then pass that QSqlQuery to another for them to call next() and value() |
| [17:29:51] | Beirdo: | that too |
| [17:30:10] | Beirdo: | I think you can count on MSqlDatabase being completely non-threadsafe |
| [17:30:13] | Beirdo: | and go from there |
| [17:30:27] | sphery: | if so, Qt-MySQL is the worst DB API ever |
| [17:30:28] | Beirdo: | anyways, I have a fix ready for the db logging |
| [17:30:35] | sphery: | (which is quite possible :) |
| [17:30:41] | Beirdo: | and will look at other candidates tonight |
| [17:33:00] | sphery: | FWIW, I'm not sure that a per-thread connection pool would be that useful--as we'd constantly have connections available for the wrong threads and be stuck waiting on connections for the right thread or be constantly opening new threads for the current thread and creating a huge pool |
| [17:33:34] | xris: | sphery: managing transactions and read state should alleviate a lot of that |
| [17:33:52] | kormoc_afk is now known as kormoc | |
| [17:37:20] | sphery: | so if everything from addDatabase() to removeDatabase() (including all QSq1Query exec() and next() and value()) has to be within the same thread, it almost sounds like it would make more sense to throw out our connection pool and make "database worker" pool so that we submit data requests to some DB request manager, which assigns tasks to the worker threads (which each have their own addDatabase("workerthread1") type connection) and then pass ... |
| [17:37:26] | sphery: | ... back results in (non-QSqlQuery containers, such as MythTV objects like ProgramInfo) |
| [17:38:06] | sphery: | in other words, maybe I need to get Beirdo to do what would be a great first step toward the MythTV dataserver TODO , now that he finished my logging rewrite TODO :) |
| [17:39:31] | Beirdo: | oh, ugh :) |
| [17:39:40] | Beirdo: | that won't be for 0.25, I'll tell ya that :0 |
| [17:40:27] | sphery: | heh |
| [17:40:59] | sphery: | what good is my TODO list if you won't finish them all up for me when I tell you to? ;) |
| [17:41:18] | Beirdo: | hehe |
| [17:41:49] | Beirdo: | well, I did get db logging to behave last night (painful debug session though) |
| [17:42:41] | Beirdo: | so as a minimum, that will be committed tonight. After that, we can hold off on the other users if there's no time, but the other candidates are: programinfo updates, scheduling, datadirect |
| [17:42:51] | Beirdo: | all of which use their own static connections |
| [17:44:12] | sphery: | but it sounds like the default connection stuff would also be affected, right? |
| [17:44:44] | sphery: | if it's only the static ones, then the connection pool won't be a problem (as the ddcon and schedcon and such connections aren't part of the normal pool) |
| [17:44:47] | Beirdo: | It could be in odd ways we don't see as often |
| [17:45:09] | Beirdo: | as we theoretically should one ever use the connection in the thread it was created in |
| [17:45:39] | Beirdo: | but as long as we remove it in hte same thread we added in, that stupid 5s delay crap shouldn't happen |
| [17:45:44] | Beirdo: | there may be other fun issues |
| [17:48:43] | sphery: | cool... so this sounds like the initial stuff will be very small/low-impact changes--and if that doesn't handle all the problems, we can consider more drastic measures |
| [17:53:50] | Beirdo: | yeah |
| [17:54:00] | Beirdo: | sorry, work keeps intruding on life :) |
| [17:54:24] | Beirdo: | it shouldn't be hard to determine which thread needs to close the static connections |
| [17:55:21] | Beirdo: | although I may need to change the programinfo updater back to a QThread (from a QRunnable IIRC) so it's a predictable thread... unless we want to close the connection after every burst of use (blech) |
IRC Logs collected by
BeirdoBot.
Please use the above link to report any bugs.