Beirdobot, irc.libera.chat

Thursday, September 30th, 2021, 00:42 UTC

[00:42:46]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has quit (Quit: Leaving.)

[02:32:35]

warpme_ (warpme_!uid391875@id-391875.helmsley.irccloud.com) has quit (Quit: Connection closed for inactivity)

[05:34:29]

Warped (Warped!~Warped@user/warped) has quit (Ping timeout: 252 seconds)

[06:10:30]

Steve-Goodey (Steve-Goodey!~steve@2a00:23c5:7d83:6501:6ea1:20d:5872:d872) has joined #mythtv

[07:55:28]

DevMythNotifyBot (DevMythNotifyBot!~nodebot@cust32-dsl91-135-4.idnet.net) has quit (Remote host closed the connection)

[07:55:35]

DevMythNotifyBot (DevMythNotifyBot!~nodebot@cust32-dsl91-135-4.idnet.net) has joined #mythtv

[08:39:17]

ooshlablu (ooshlablu!~ooshlablu@2601:18d:4600:43f0:a038:14ff:fe50:8510) has quit (Ping timeout: 246 seconds)

[08:55:58]

DevMythNotifyBot (DevMythNotifyBot!~nodebot@cust32-dsl91-135-4.idnet.net) has quit (Remote host closed the connection)

[08:56:11]

DevMythNotifyBot (DevMythNotifyBot!~nodebot@cust32-dsl91-135-4.idnet.net) has joined #mythtv

[08:59:58]

knowledgejunkie (knowledgejunkie!~knowledge@cpc77018-warw18-2-0-cust959.3-2.cable.virginm.net) has quit (Ping timeout: 260 seconds)

[09:06:50]

knowledgejunkie (knowledgejunkie!~knowledge@cpc77018-warw18-2-0-cust959.3-2.cable.virginm.net) has joined #mythtv

[09:21:08]

Steve-Goodey (Steve-Goodey!~steve@2a00:23c5:7d83:6501:6ea1:20d:5872:d872) has quit (Quit: Konversation terminated!)

[10:16:09]

paul-h (paul-h!~paul@pres-25-b2-v4wan-161467-cust2666.vm29.cable.virginm.net) has joined #mythtv

[10:16:09]

paul-h (paul-h!~paul@pres-25-b2-v4wan-161467-cust2666.vm29.cable.virginm.net) has quit (Changing host)

[10:16:09]

paul-h (paul-h!~paul@mythtv/developer/paul-h) has joined #mythtv

[10:16:09]

Mode for #mythtv by ChanServ!ChanServ@services.libera.chat : +v paul-h

[10:16:19]

** stuarta waves to paul-h **

[10:16:48]

paul-h:

Morning Stuart

[10:25:49]

DevMythNotifyBot:

issue opened by stuarta, "pcre2 JIT regression causing unit test failures": https://github.com/MythTV/mythtv/issues/395

[10:25:58]

DevMythNotifyBot:

issue assigned by stuarta, "pcre2 JIT regression causing unit test failures": https://github.com/MythTV/mythtv/issues/395

[10:26:31]

DevMythNotifyBot:

issue labeled by stuarta, "pcre2 JIT regression causing unit test failures": https://github.com/MythTV/mythtv/issues/395 Labels: "version:master" "version:v31-fixes" "upstream"

[10:26:31]

DevMythNotifyBot:

issue labeled by stuarta, "pcre2 JIT regression causing unit test failures": https://github.com/MythTV/mythtv/issues/395 Labels: "version:master" "version:v31-fixes" "upstream"

[10:26:31]

DevMythNotifyBot:

issue labeled by stuarta, "pcre2 JIT regression causing unit test failures": https://github.com/MythTV/mythtv/issues/395 Labels: "version:master" "version:v31-fixes" "upstream"

[10:29:40]

stuarta:

paul-h: fyi ^^^ that affects ubuntu-21.10

[10:30:35]

paul-h:

OK

[10:30:52]

stuarta:

it's been fixed in rawhide and arch, but the beta is affected

[10:35:35]

ooshlablu (ooshlablu!~ooshlablu@2601:18d:4600:43f0:a038:14ff:fe50:8510) has joined #mythtv

[10:38:57]

paul-h:

Where do we use it apart from the unit tests?

[10:40:14]

stuarta:

all the eitfixups....

[10:40:29]

stuarta:

well not all 2/40 fail

[10:40:58]

stuarta:

but potentially anywhere that sort of regexp is used will stop working

[10:51:03]

paul-h:

Getting lots of 'server replied: Service Unavailable errors from the new API server :(

[10:51:24]

stuarta:

interesting, we need to track that down

[10:51:54]

paul-h:

The same url works fine in Firefox but fails in my app that uses the QT networking

[10:52:28]

stuarta:

grab the --verbose http logs and see what it says (i don't *think* you need --verbose http:debug)

[10:52:38]

stuarta:

from the backend

[10:55:39]

paul-h:

I'm wondering if the server can't handle concurrent requests properly since the ones that fail follow each other in rapid succession like grabbing channel icons to display in a grid for example

[10:56:05]

stuarta:

mark said it should, but lets do the analysis and find out

[10:56:42]

stuarta:

maybe the actual v2 services need extra work

[11:02:50]

stuarta:

paul-h: so the first call is working, just later ones (or a selection of later ones) aren't working

[11:02:53]

stuarta:

?

[11:04:12]

paul-h:

Look like the server only allows 4 threads "E HTTPServer http/mythhttpserver.cpp:258 (newTcpConnection) HTTPServer: No HTTP threads available (Have: 0 Max: 4)"

[11:04:27]

stuarta:

so it needs to queue the requests

[11:05:22]

stuarta:

curious the qt libs don't reuse the existing tcp connection

[11:06:48]

paul-h:

I'm going to get a bite to eat back later

[11:06:52]

stuarta:

np

[11:25:50]

stuarta:

hampton: installed soundtouch dev packages on all my builders. centos7 is the only odd one out with v1.4, the rest are later

[12:20:42]

Steve-Goodey (Steve-Goodey!~steve@2a00:23c5:7d83:6501:6ea1:20d:5872:d872) has joined #mythtv

[13:02:32]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has joined #mythtv

[13:02:32]

Mode for #mythtv by ChanServ!ChanServ@services.libera.chat : +v peterbennett

[13:16:23]

paul-h:

stuarta: I could be wrong but it looks to me like the server can only handle a maximum of 4 requests at a time

[13:16:59]

paul-h:

it does send a Retry-After: 10 header when it fails

[13:17:26]

stuarta:

equally it could accept and then queue the connection for an available worker

[13:18:54]

paul-h:

I think that makes more sense in a API that is going to be used by a GUI app. A 10 second delay is too long

[13:21:29]

paul-h:

It doesn't help that my Samsung TV appears to be querying the backend for UPnP stuff at the same time which I guess is using one of the threads

[13:21:32]

stuarta:

not to mention random

[13:21:54]

stuarta:

no, upnp still uses the old stack, it's not been ported yet

[13:27:32]

paul-h:

Do you know how the old server works does it queue the requests?

[13:27:52]

stuarta:

not off the top of my head, i would have to investigate

[13:42:10]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has quit (Remote host closed the connection)

[13:43:34]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has joined #mythtv

[13:43:34]

Mode for #mythtv by ChanServ!ChanServ@services.libera.chat : +v peterbennett

[13:47:02]

peterbennett:

paul-h: I am surprised you are getting errors from teh new service. I am able to run leanfront without problem and it uses a lot of calls. Is it one particular api call that fails?

[13:57:47]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has quit (Quit: Leaving.)

[14:04:31]

paul-h:

They all fail randomly

[14:06:51]

paul-h:

Probably because Qt QML's networking is asynchronous so the xml model stuff and image items can fire lots of requests at once

[14:07:01]

stuarta:

paul-h: can we get it down to a reproducer with curl?

[14:10:09]

paul-h:

If my theory is correct just fire more that 4 requests at the server and it will fail

[14:15:36]

stuarta:

paul-h: which service?

[14:19:43]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has joined #mythtv

[14:19:43]

Mode for #mythtv by ChanServ!ChanServ@services.libera.chat : +v peterbennett

[14:20:03]

paul-h:

All of them I think the ones I can see right now are Dvr/GetEncoderList, Channel/GetChannelInfoList, Guide/GetChannelIcon

[14:21:26]

paul-h:

I'm just going to go back to the old server for now the new one is unusable for me :(

[14:22:01]

stuarta:

thanks for testing

[14:22:29]

stuarta:

will let you know if we have something extra to test

[14:26:51]

paul-h:

OK thanks it's a pity there are not more users of the API who can test this to make sure it is reproducible on other systems

[14:29:44]

paul-h:

If Peter is correct and it works for him then I have a nagging suspicion it is something to do with the way Qt's QML networking works

[14:30:10]

stuarta:

it certainly sounds like it's not pipelining and reusing the connections

[14:30:27]

peterbennett:

I just ran 6 calls at once – no error

[14:30:39]

peterbennett:

They were the same call though

[14:31:08]

peterbennett:

for x in 1 2 3 4 5 6 ; do curl "http://rocinante:6744/Guide/GetProgramGuide?S . . . 00:00Z" > out$x & done

[14:31:24]

stuarta:

i was going to use `ab`

[14:31:41]

peterbennett:

all 6 output files are the same 500k of data

[14:32:13]

peterbennett:

I am not familiar with ab

[14:32:28]

stuarta:

it's long name is apache bench, used to load test apache

[14:32:35]

stuarta:

or any other webserver

[14:32:46]

peterbennett:

paul-h: If you can give me your program I can test it and fix what is necessary

[14:33:58]

peterbennett:

By the way there are some spurious messages from the new server, for example "MetaMethod: Method 'GetProgramList' takes more than 9 parameters; will probably fail"

[14:34:24]

peterbennett:

This is incorrect, services work fine with as many paramaters as you like, I have tested it

[14:37:33]

paul-h:

when you run that script can you check the backend log and see if they are all using the same thread? They are named HTTP0 to HTTP3 and the log messages all show which thread is in use

[14:37:49]

stuarta:

peterbennett: i remember mark saying that according to the docs it's not guaranteed to work, but it did in all his testing

[14:38:07]

stuarta:

also i believe the max threads is controlled by your cpu count

[14:38:22]

peterbennett:

paul-h: d what log level are you using? All I see is "I HTTPService: Deleting object"

[14:38:47]

stuarta:

yup -> "WebSocketServer(): Max Thread Count 16" <- i have 8 cpus with hyperthreading

[14:38:52]

peterbennett:

The code is limiting it by the cpu count?

[14:39:03]

stuarta:

the thread count yes

[14:39:06]

paul-h:

peterbennett: just with http

[14:39:38]

peterbennett:

Ok wherever it does that just change it to hardcode 20 or so.

[14:40:22]

stuarta:

that's not a proper solution, that's a workaround

[14:40:29]

stuarta:

and you can still potentially hit it

[14:40:57]

paul-h:

I run my BE in a VM which I think has 4 cpus

[14:41:23]

peterbennett:

I have 8 cpus on this laptop so the 6 count is not a good test

[14:42:41]

paul-h:

Actually no only 2 cpu's but I see see it allocates 4 threads to the http server

[14:42:44]

stuarta:

here you go -> ab -n 1000 -c 100 http://localhost:7544/Channel/GetChannelInfoList

[14:42:54]

stuarta:

adjust as necessary

[14:43:09]

peterbennett:

Time to fire up my old slow system that has only 2 cpus ?

[14:43:54]

stuarta:

peterbennett: -c 100 in the above command is the concurrency

[14:44:00]

peterbennett:

Where will the output go from that ab?

[14:44:07]

stuarta:

1000 requests 100 in parallel.

[14:44:12]

stuarta:

ab eats the output by default

[14:44:22]

peterbennett:

need to pipe it to files to check if any have failure messages

[14:45:27]

stuarta:

Concurrency Level: 100

[14:45:27]

stuarta:

Time taken for tests: 11.941 seconds

[14:45:27]

stuarta:

Complete requests: 1000

[14:45:27]

stuarta:

Failed requests: 0

[14:45:32]

peterbennett:

isn't it 6744 not 7544 ?

[14:45:35]

stuarta:

^^^ tells you none failed

[14:45:44]

stuarta:

peterbennett: yes by default, that's where my test backend lives

[14:46:26]

peterbennett:

I see a bunch of errors HTTPServer: No HTTP threads available (Have: 0 Max: 16)

[14:47:11]

peterbennett:

OK I guess everything works fro me because I have 16 threads

[14:47:49]

stuarta:

should be able to use cpuset to limit cpu's visible to the process

[14:47:57]

** stuarta tests **

[14:49:02]

peterbennett:

My test failed with apr_socket_recv: Connection reset by peer (104) after all those no threads available messages

[14:50:32]

peterbennett:

This server is not supposed to be high-volume many-user, so setting max threads to something like 16 or letting it be a setting in the database seems a reasonable compromise.

[14:50:51]

stuarta:

i'd still like to understand why it occurs

[14:51:43]

peterbennett:

I only have 8 processors but I get 16 threads

[14:52:01]

stuarta:

same here, HT normally does that

[14:55:01]

peterbennett:

With my test with the "for" statement – if I increase to 18, then two of them fail

[14:55:09]

peterbennett:

So it is consistent

[14:56:46]

peterbennett:

My CPU has 4 cores – 8 with hyperthreading, slo that does not explain the 16

[14:57:17]

peterbennett:

Intel® Core™ i5–9300H Processor

[14:58:04]

stuarta:

weird

[14:58:20]

peterbennett:

I suspect the code is taking the number of cpus and multiplying by 2

[14:58:26]

paul-h:

There is this in MythHTTPThreadPool m_maxThreads = static_cast<size_t>(std::max(QThread::idealThreadCount() * 2, 4));

[14:59:22]

peterbennett:

Since these are not cpu-bound threads it is ok to go more than that

[15:00:02]

peterbennett:

I would suggest a minimum of 16 rather than 4

[15:01:15]

peterbennett:

or better still – a setting in the database than can be changed if needed by somebody with an unusual pattern of calls

[15:02:56]

stuarta:

wish i could reproduce it!

[15:03:05]

peterbennett:

paul-h: Does your code do multiple simultaneous calls?

[15:05:09]

peterbennett:

I think the only time I may be doing that is when it is loading images for a list of shows

[15:06:08]

peterbennett:

stuarta: You need a slower machine!

[15:06:13]

stuarta:

:-p

[15:06:36]

stuarta:

i have one of those too. going to kick off a build and wait 45mins

[15:06:45]

paul-h:

It's my QML frontend I'm testing which just uses the normal Qt QML networking that is always asynchronous so yes there can be multiple simultaneous calls

[15:08:35]

stuarta:

load average: 102.17, 74.03, 35.83

[15:08:37]

stuarta:

:)

[15:09:02]

stuarta:

running 10k request, 100 concurrency against Channel/GetChannelInfoList

[15:09:19]

stuarta:

and it still aint breaking :-/

[15:09:43]

peterbennett:

how many channels you got?

[15:10:42]

stuarta:

702

[15:10:48]

peterbennett:

I have several hundred

[15:11:54]

stuarta:

mine's an m3u file to define a bunch of iptv streams, handy for testing without a tuner

[15:12:00]

peterbennett:

SELECT count(*) from channel; gives 737

[15:13:29]

stuarta:

curl -H 'Accept: application/json' http://localhost:7544/Channel/GetChannelInfoList | jq '.ChannelInfoList.Count'

[15:13:39]

stuarta:

gives 702

[15:14:23]

peterbennett:

that gives 737 for me

[15:14:47]

stuarta:

good to see the api gives the same value as a select from the DB

[15:28:41]

stuarta:

https://github.com/MythTV/mythtv/blob/devel/h . . . pp#L258-L263

[15:28:57]

stuarta:

i'm not sure the comment on line 261 reflects reality

[15:29:26]

stuarta:

so that's where we can enqueue requests

[15:29:29]

stuarta:

then elsewhere, the threads need to look for work

[15:38:12]

peterbennett:

I don't understand why the web server all use a pool of threads rather than just creating them when needed and deleting them when not needed.

[15:38:36]

stuarta:

that's what it does do, creates them when needed

[15:38:53]

stuarta:

the code above the highlighted bit in the above link does that

[15:39:11]

peterbennett:

Ok so then why have a maximum?

[15:39:40]

stuarta:

DOS protection, should always have a limit

[15:39:47]

peterbennett:

Does it delet the thread when finished?

[15:40:11]

stuarta:

i believe the thread exits when it's finished the work

[15:40:17]

** stuarta is unpicking it **

[15:40:38]

peterbennett:

So for DOS protection you don't want to queue them, the current logic is correct but just raise the limit

[15:40:54]

stuarta:

i think there's a happy medium

[15:41:05]

peterbennett:

If you just queu them then DOS will still bring you down evetntually

[15:41:17]

stuarta:

it may be that setting a better minimum would be more appropriate

[15:41:38]

stuarta:

bah there's the code that i needed to find

[15:42:45]

stuarta:

https://github.com/MythTV/mythtv/blob/devel/h . . . pool.cpp#L15 <- sets max pending = num thread

[15:42:50]

stuarta:

*threads

[15:42:54]

stuarta:

it should be a multiple

[15:44:20]

stuarta:

paul-h: set the number in the above link to a multiple of m_maxThreads and see how you get on

[15:45:39]

stuarta:

and i also think peterbennett is right, we should increase min threads to 16

[15:53:02]

stuarta:

i wonder if i force maxthreads to 2 i can reproduce it....

[15:54:55]

stuarta:

nope, this machine is too fast

[16:00:01]

stuarta:

reading the comment at mythhttpthreadpool.cpp:13–14 it's explicitly suggesting not to allow more connections than we have threads because otherwise browsers consume them

[16:06:14]

stuarta:

on the other hand, the hp microserver gen 8, is much slower, and still building the code :-(

[16:10:50]

paul-h:

stuarta: I've changed it to be m_maxThreads x 4 which should be 16 threads in my case, it could be a few hours before Launchpad has the build ready for me to test though

[16:15:53]

paul-h:

The problem I have with increasing the thread count is it's not really fixing the problem it's just making it less likely to happen

[16:16:21]

paul-h:

It's a lot easier that creating a queue though :)

[16:17:35]

paul-h:

I seem to remember Mark commenting he wanted the server it to be lockless so maybe that is why he did it this way

[16:17:46]

peterbennett:

There has to be some sort of limit, as stuart said, to prevent DOS attacks. It just needs to be high enough.

[16:19:06]

peterbennett:

Since I do not expect DOS attacks in my house I would be fine with setting the limit very high.

[16:20:05]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has quit (Quit: Leaving.)

[16:22:56]

paul-h:

You have also got to think about not bringing the BE to it's knees handling Service API requests which could corrupt recordings

[16:24:10]

paul-h:

I go back to what I have been saying for years that server should not be part of the backend it should be a separate server

[16:25:07]

paul-h:

If you bring down the web server you also bring down all the recordings

[16:25:48]

troyt (troyt!~troyt@2601:681:4100:d591:44dd:acff:fe85:9c8e) has quit (Ping timeout: 250 seconds)

[16:47:04]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has joined #mythtv

[16:47:04]

Mode for #mythtv by ChanServ!ChanServ@services.libera.chat : +v peterbennett

[17:07:47]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has quit (Quit: Leaving.)

[17:10:35]

paul-h_ (paul-h_!~paul@pres-25-b2-v4wan-161467-cust2666.vm29.cable.virginm.net) has joined #mythtv

[17:11:09]

knowledg1junkie (knowledg1junkie!~knowledge@cpc77018-warw18-2-0-cust959.3-2.cable.virginm.net) has joined #mythtv

[17:17:36]

ooshlablu (ooshlablu!~ooshlablu@2601:18d:4600:43f0:a038:14ff:fe50:8510) has quit (*.net *.split)

[17:17:36]

paul-h (paul-h!~paul@mythtv/developer/paul-h) has quit (*.net *.split)

[17:17:36]

knowledgejunkie (knowledgejunkie!~knowledge@cpc77018-warw18-2-0-cust959.3-2.cable.virginm.net) has quit (*.net *.split)

[17:17:37]

poptix (poptix!poptix@poptix.net) has quit (*.net *.split)

[17:17:51]

paul-h_ is now known as paul-h

[17:21:32]

paul-h (paul-h!~paul@pres-25-b2-v4wan-161467-cust2666.vm29.cable.virginm.net) has quit (Remote host closed the connection)

[17:21:46]

paul-h (paul-h!~paul@pres-25-b2-v4wan-161467-cust2666.vm29.cable.virginm.net) has joined #mythtv

[17:21:46]

paul-h (paul-h!~paul@pres-25-b2-v4wan-161467-cust2666.vm29.cable.virginm.net) has quit (Changing host)

[17:21:46]

paul-h (paul-h!~paul@mythtv/developer/paul-h) has joined #mythtv

[17:21:46]

Mode for #mythtv by ChanServ!ChanServ@services.libera.chat : +v paul-h

[17:22:10]

ooshlablu (ooshlablu!~ooshlablu@2601:18d:4600:43f0:a038:14ff:fe50:8510) has joined #mythtv

[17:23:00]

poptix (poptix!poptix@poptix.net) has joined #mythtv

[17:43:18]

MythBuild_ (MythBuild_!~MythBuild@mizar.mythtv.org) has joined #mythtv

[17:44:29]

MythBuild (MythBuild!~MythBuild@mizar.mythtv.org) has quit (Ping timeout: 265 seconds)

[17:55:50]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has joined #mythtv

[17:55:50]

Mode for #mythtv by ChanServ!ChanServ@services.libera.chat : +v peterbennett

[18:43:18]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has quit (Quit: Leaving.)

[19:01:53]

warpme_ (warpme_!uid391875@id-391875.helmsley.irccloud.com) has joined #mythtv

[19:03:41]

warpme_:

huh: qt6 is at 6.2 and will be lts. to cross-compile first i need to build 4679 native .o objects. "good" cross-compile approach. very good........

[20:03:58]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has joined #mythtv

[20:03:58]

Mode for #mythtv by ChanServ!ChanServ@services.libera.chat : +v peterbennett

[20:19:52]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has quit (Quit: Leaving.)

[20:39:00]

Steve-Goodey (Steve-Goodey!~steve@2a00:23c5:7d83:6501:6ea1:20d:5872:d872) has quit (Quit: Konversation terminated!)

[20:55:06]

DevMythNotifyBot:

COMMIT: linuxdude42 pushed 1 commit(s) to branch master: https://github.com/MythTV/mythtv/compare/63bb . . . f3ad85001cc7

[21:00:45]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has joined #mythtv

[21:00:45]

Mode for #mythtv by ChanServ!ChanServ@services.libera.chat : +v peterbennett

[21:17:35]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has quit (Quit: Leaving.)

[21:17:59]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has joined #mythtv

[21:17:59]

Mode for #mythtv by ChanServ!ChanServ@services.libera.chat : +v peterbennett

[22:00:42]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has quit (Quit: Leaving.)

[22:17:03]

paul-h (paul-h!~paul@mythtv/developer/paul-h) has quit (Quit: Konversation terminated!)

[22:33:04]

amessina (amessina!~amessina@user/amessina) has joined #mythtv

[22:56:29]

amessina (amessina!~amessina@user/amessina) has quit (Quit: Konversation terminated!)

[23:42:46]

peterbennett (peterbennett!~peter@mythtv/developer/peterbennett) has joined #mythtv

[23:42:46]

Mode for #mythtv by ChanServ!ChanServ@services.libera.chat : +v peterbennett

MythLogBot@irc.libera.chat :: #mythtv

Current users (27):