diff --git a/devdocs/todo.txt b/devdocs/todo.txt index 3b054baa..387d7c0b 100644 --- a/devdocs/todo.txt +++ b/devdocs/todo.txt @@ -12,6 +12,15 @@ todo: OPS routes (SERVER AND CLIENT) BACKUP & related stuff that should be in v.initial release https://rockfish.ayanova.com/default.htm#!/rfcaseEdit/3369 todo: OPS notification created for failed jobs also maybe direct immediate email bypassing generator? + +todo: Search indexing performance improvement and exception avoidance (Search.cs 828) + ON CONFLICT IDEA + https://www.postgresql.org/docs/current/sql-insert.html#SQL-ON-CONFLICT + Idea: do the insert manually with the clause "on conflict do nothing" + if detect it hasn't inserted (conflict) trigger a fetch instead + like what is being done now but won't have the exception to deal with!! + + todo: (BREAK THIS OUT INTO LATER/NOW/CASES) there are several outstanding AUTHENTICATION related cases in rockfish for RAVEN e.g. https://rockfish.ayanova.com/default.htm#!/rfcaseEdit/1924 https://rockfish.ayanova.com/default.htm#!/rfcaseEdit/1835 diff --git a/server/AyaNova/biz/Search.cs b/server/AyaNova/biz/Search.cs index 391a9a5e..2bc1327a 100644 --- a/server/AyaNova/biz/Search.cs +++ b/server/AyaNova/biz/Search.cs @@ -792,6 +792,11 @@ Now going to try the opposite, a *lot* of text 10 paragraphs in both c2 and note So the quantity of text directly affects the performance, so it's not just some overhead from the query being run, it's the amount of work it needs to do in the queries +cache or provide directly the translation to save time repeatedly fetching it when doing bulk ops!!! + -After doing this 500=21 That's as fast as when I stripped out all the text, what a huge overhead saving right there!: + 2020-01-24 12:00:41.2547|INFO|Seeder|Seeding 500 Widgets.... + 2020-01-24 12:00:51.9138|INFO|Seeder|500 Widgets seeded in 10649 ms + THINGS TO TRY: Completely alternate methods: - https://stackoverflow.com/a/15089664/8939 Store a Digest of each record with that record then can just search the digests (would mean a search has to traverse all records of every table possibly) @@ -804,10 +809,14 @@ Async the keyword processing Removing use of EF entirely in search indexing processing in favor of direct sql queries -cache or provide directly the translation to save time repeatedly fetching it when doing bulk ops!!! - -After doing this 500=21 That's as fast as when I stripped out all the text, what a huge overhead saving right there!: - 2020-01-24 12:00:41.2547|INFO|Seeder|Seeding 500 Widgets.... - 2020-01-24 12:00:51.9138|INFO|Seeder|500 Widgets seeded in 10649 ms +Transaction locked faster than exception method now used? +Make indexing a job that happens periodically behind the scenes based on last indexing date and event log? + +ON CONFLICT IDEA + https://www.postgresql.org/docs/current/sql-insert.html#SQL-ON-CONFLICT + Idea: do the insert manually with the clause "on conflict do nothing" + if detect it hasn't inserted (conflict) trigger a fetch instead + like what is being done now but won't have the exception to deal with!! ///////////////////////////////////////////////////////////////////////////////