Commit Graph

746 Commits

Author SHA1 Message Date
jygaulier
27433c769f PHRAS-3412_bad-gps-crash-indexation_MASTER
fix : PHRAS-3412 bad gps values are ignored
2021-04-01 15:45:40 +02:00
aina esokia
957f9712ad PHRAS-3279 2020-12-10 14:24:22 +03:00
aina esokia
36f2e1a640 PHRAS-3279 2020-12-09 17:13:32 +03:00
aina esokia
c1090021dd fix 2020-12-09 17:00:49 +03:00
aina esokia
f411c715e3 PHRAS-3279 add populate_permalinks key in configuration 2020-12-09 16:48:07 +03:00
jygaulier
889471a3ff add : don't create missing permalinks during indexation (PHRAS-3279) 2020-11-24 17:23:07 +01:00
Nicolas Maillat
7b25a719b7 PHRAS-3658 enabled path 2020-11-20 10:28:19 +01:00
jygaulier
a5b2a3cc14 add : permalinks are injected in es (PHRAS-3279) 2020-11-19 21:59:29 +01:00
aina esokia
dd3784aebd PHRAS-3265 taken account max_result_window 2020-11-18 12:15:03 +03:00
jygaulier
53e116cf4f fix : conf:max_result_window (es) is now used when creating index by searchengine 2020-11-13 10:10:35 +01:00
aina esokia
908ec3c0a1 PHRAS-3265 port max_result_window 2020-11-11 15:16:25 +03:00
jygaulier
cfd385498d - add : cache on es "tags"
- removed : variable globalstructure construction (WITH_FIELDS, WITH_FLAGS...) (was unused)
- cleanup (stopwatches etc)
2020-11-05 19:54:14 +01:00
jygaulier
5ae105d9c2 - fix : error during populate
- add : clear structure cache when structure (field or sb) is changed
2020-11-05 11:27:39 +01:00
jygaulier
4d961e2dbe - fix : added cache to es "globalstructure" construction 2020-11-04 11:35:36 +01:00
jygaulier
e36b0785ee - add some simple cache
- disable check of conf/searchengine/type
- faster construction of concept-paths
- pass strucure to es as factory (useless, to be reverted)
- add stopwatch (log into file, disabled for now)
2020-10-30 11:00:17 +01:00
jygaulier
b0adb104d1 - remove "path" from subdefs 2020-10-27 19:30:11 +01:00
jygaulier
7c01c34dfc - add timing debug, disable permalinks injection to es 2020-10-27 17:18:37 +01:00
jygaulier
c3ffbd2820 WIP / DEBUG
- ignore failing permalinks during es indexation
2020-10-21 20:31:20 +02:00
jygaulier
ff44cc379e WIP / DEBUG
- add every subdef in es document, including permalink
2020-10-21 20:07:11 +02:00
jygaulier
d0283913de WIP / DEBUG
- add stopwatches to find slow parts.
- to test : add api-v3/searchraw ; same as "search", but returns raw es docs (shortest way). No objects, no fractal.
2020-10-21 20:07:11 +02:00
aina esokia
585a24c40d PHRAS-3224 2020-10-01 15:11:54 +03:00
jygaulier
e79f2cb4f2 add:
- multiple pagination methods
- "include" parameter in /search route :
-- facets : facets are no more returned by default
-- results.stories.children : to populate the children of stories
- "story_children_limit" parameter in /search route
  will limit the number of children populated by story
- "count" (for results) and "children_count" (for stories in results) is the number of items in the current page
- "total" (for results) and "children_total" (for stories in results) is the total number of items

todo: swagger doc (but impossible to describe related/excluded parameters (like pagination methods in /search)
2020-09-22 21:30:19 +02:00
jygaulier
71dd539c76 fix : if string cutting at 32766 OCTETS for es/lucene happened in the middle of a utf8 CHAR, string was invalid. 2020-07-06 18:06:51 +02:00
Jean-Yves Gaulier
9b202bea31 PHRAS-3023_slow-query-while-indexing_4.1
- fix break with "bin/developer phraseanet:regenerate-sqlite" due to column add in sql result
2020-04-09 21:47:22 +02:00
Jean-Yves Gaulier
ff41be7e0e PHRAS-3023_slow-query-while-indexing_4.1
- fix rid-list fetcher
2020-04-09 19:57:21 +02:00
Jean-Yves Gaulier
4350e0adf0 PHRAS-3023_slow-query-while-indexing_4.1
- fix php 7.2 only syntax
2020-04-09 19:46:25 +02:00
Jean-Yves Gaulier
e1a3535afa PHRAS-3023_slow-query-while-indexing_4.1
- revert php 7.2 only syntax
2020-04-09 19:23:17 +02:00
Jean-Yves Gaulier
cb6a781b02 PHRAS-3023_slow-query-while-indexing_4.1
- fix bound limit depending on order/direction options
- add index on dbox:record.moddate
2020-04-09 19:11:08 +02:00
Nicolas Maillat
45dfa8050f Merge branch 'master' into PHRAS-3015_sb31-crashes-indexer_4.1 2020-04-08 23:39:04 +02:00
Jean-Yves Gaulier
3b99f5bb50 PHRAS-3023_slow-query-while-indexing_4.1
fix : remove double where
2020-04-08 20:06:19 +02:00
Jean-Yves Gaulier
2b4accae91 PHRAS-3023_slow-query-while-indexing_4.1
fix : change fetcher query to paginate with record_id/count (not limit/count)
2020-04-08 19:50:57 +02:00
Jean-Yves Gaulier
662fe95712 PHRAS-3015_sb31-crashes-indexer_4.1
fix : set es:flags_bitfield as long (signed 64 bits)
2020-04-06 19:51:52 +02:00
jygaulier
5df663cad2 Merge branch 'master' into PHRAS-3001_es-geopoint-sources_4.1 2020-03-31 17:54:55 +02:00
Jean-Yves Gaulier
027f7e1571 PHRAS-3001_es-geopoint-sources_4.1
- fix : exif source is a fallback if no good values are found in the configured source field (or if conf is not set)
2020-03-31 17:54:22 +02:00
Jean-Yves Gaulier
9f216813b7 PHRAS-3001_es-geopoint-sources_4.1
- fix : follow the original conf structure (position-fields into provider)
- add : handles "latlng" format
2020-03-31 17:16:57 +02:00
Jean-Yves Gaulier
40b9a7cd0c PHRAS-3003_large-value-32ko_es_4.1
- fix (temporary ?) : text values are kept full in metadata table, but truncated to 32Ko by indexer.
2020-03-30 15:46:04 +02:00
Jean-Yves Gaulier
8ffeb58289 PHRAS-3001_es-geopoint-sources_4.1
WIP
- fix : gps position indexed by es now comes from the metadata fields defined in conf; (not from exif anymore).
- todo/wip : decide conf place, since today we only have an array of geo providers
- todo/wip : does exif apply as fallback if nothing in conf ? or if metadata not set in record ?
2020-03-29 20:11:03 +02:00
Jean-Yves Gaulier
722b1e3750 include Harrys's new templates
fix form restore
fix Version
2020-02-17 20:17:08 +01:00
Jean-Yves Gaulier
4f7be4c0a9 fix bad merge happened cause last rebase (?) 2020-02-17 20:17:08 +01:00
Jean-Yves Gaulier
eeec1ab716 save facets following the order of the admin form. 2020-02-17 20:17:08 +01:00
Jean-Yves Gaulier
3b8d75d39d fix : tech facets did appear twice. 2020-02-17 20:17:08 +01:00
Jean-Yves Gaulier
b1b35cc92b fix malformed query on tech fields ('_' in query) 2020-02-17 20:17:08 +01:00
Jean-Yves Gaulier
20839f1dd7 return facets following the order defined in conf. 2020-02-17 20:17:08 +01:00
Jean-Yves Gaulier
53f17e1897 save modifications in conf, and also write back to databoxes 2020-02-17 20:17:08 +01:00
Jean-Yves Gaulier
8171b7c1f1 PHRAS-2879_facet-order_4.1
fix setup:install (cont' 2)
2020-02-17 20:17:08 +01:00
Jean-Yves Gaulier
4305441a03 PHRAS-2879_facet-order_4.1
fix setup:install (cont')
2020-02-17 20:17:08 +01:00
Jean-Yves Gaulier
6838198a69 PHRAS-2879_facet-order_4.1
fix setup:install.
2020-02-17 20:17:08 +01:00
Jean-Yves Gaulier
2ccdcfed27 PHRAS-2879_facet-order_4.1
save all facet settings in conf, order is preserved in admin/search-engine-settings.
todo : ux to re-order facets
todo : return facets into query results in this setting order.
todo : migration from dbox settings (field struct) to conf ; remove field setting from admin
2020-02-17 20:17:08 +01:00
jygaulier
a2b2658c04 Merge branch 'master' into PHRAS-2831_port-truncation_4.1 2019-11-19 17:34:45 +01:00
Jean-Yves Gaulier
3145049247 PHRAS-2831_port-truncation_4.1
copy elastic/truncation code from 4.0
nb: from already exists
2019-11-19 16:45:46 +01:00