summary refs log tree commit diff
diff options
context:
space:
mode:
authorJune McEnroe <june@causal.agency>2020-01-14 17:57:04 -0500
committerJune McEnroe <june@causal.agency>2020-01-14 17:57:04 -0500
commit1140dfdd3d6bbcadb0ccb1693b4fbe4e6d0792c5 (patch)
tree804d2897a5be3c12b50d5e723f40aeadc5e8fdb5
parentAdd option for client cert and SASL EXTERNAL (diff)
downloadlitterbox-1140dfdd3d6bbcadb0ccb1693b4fbe4e6d0792c5.tar.gz
litterbox-1140dfdd3d6bbcadb0ccb1693b4fbe4e6d0792c5.zip
Add columnsize = 0 option to FTS index
Since we don't use ranking functions and I don't see them being useful,
there is no point in having columnsize, which just takes extra space in
the database.

In my database of approximately 3.5 million events, disabling columnsize
saves about 62 MB.

The migration unfortunately has to rebuild the entire index to disable
it.
-rw-r--r--database.h25
1 files changed, 22 insertions, 3 deletions
diff --git a/database.h b/database.h
index d44bb15..83be73c 100644
--- a/database.h
+++ b/database.h
@@ -31,7 +31,7 @@
 
 #define DATABASE_PATH "litterbox/litterbox.sqlite"
 
-enum { DatabaseVersion = 0 };
+enum { DatabaseVersion = 1 };
 
 #define ENUM_TYPE \
 	X(Privmsg, "privmsg") \
@@ -285,7 +285,8 @@ static const char *InitSQL = SQL(
 		network, channel, query, nick, user, target, message,
 		content = text,
 		content_rowid = event,
-		tokenize = 'porter'
+		tokenize = 'porter',
+		columnsize = 0
 	);
 
 	CREATE TRIGGER eventsInsert AFTER INSERT ON events BEGIN
@@ -300,6 +301,8 @@ static const char *InitSQL = SQL(
 		) SELECT 'delete', * FROM text WHERE event = old.event;
 	END;
 
+	PRAGMA user_version = 1;
+
 	COMMIT TRANSACTION;
 );
 
@@ -308,7 +311,23 @@ static inline void dbInit(void) {
 }
 
 static const char *MigrationSQL[] = {
-	NULL,
+	// Added columnsize = 0 option.
+	SQL(
+		BEGIN TRANSACTION;
+		DROP TABLE search;
+		CREATE VIRTUAL TABLE search USING fts5 (
+			network, channel, query, nick, user, target, message,
+			content = text,
+			content_rowid = event,
+			tokenize = 'porter',
+			columnsize = 0
+		);
+		INSERT INTO search (
+			rowid, network, channel, query, nick, user, target, message
+		) SELECT * FROM text;
+		PRAGMA user_version = 1;
+		COMMIT TRANSACTION;
+	),
 };
 
 static inline void dbMigrate(void) {
n A. Donenfeld <Jason@zx2c4.com> 2014-01-17makefile: bump versionJason A. Donenfeld Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> 2014-01-17mailmap: source before lighttpdJason A. Donenfeld Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> 2014-01-17ui-shared: do not allow negative minutesJason A. Donenfeld Do to timestamp differences, sometimes cgit would should "-0 min", which doesn't make any sense. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> 2014-01-17auth: document tweakables in lua scriptJason A. Donenfeld Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> 2014-01-17repolist: make owner clickable to searchJason A. Donenfeld Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> 2014-01-17ui-shared: move about tab all the way to the leftJason A. Donenfeld There were no objections (at the time of committing this): http://lists.zx2c4.com/pipermail/cgit/2013-May/001393.html http://lists.zx2c4.com/pipermail/cgit/2014-January/001904.html Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> 2014-01-17filter: don't forget to reap the auth filterJason A. Donenfeld Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> 2014-01-17cgit.c: free tmp variableJason A. Donenfeld Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> 2014-01-17Switch to exclusively using global ctxLukas Fleischer Drop the context parameter from the following functions (and all static helpers used by them) and use the global context instead: * cgit_print_http_headers() * cgit_print_docstart() * cgit_print_pageheader() Remove context parameter from all commands Drop the context parameter from the following functions (and all static helpers used by them) and use the global context instead: * cgit_get_cmd() * All cgit command functions. * cgit_clone_info() * cgit_clone_objects() * cgit_clone_head() * cgit_print_plain() * cgit_show_stats() In initialization routines, use the global context variable instead of passing a pointer around locally. Remove callback data parameter for cache slots This is no longer needed since the context is always read from the global context variable. Signed-off-by: Lukas Fleischer <cgit@cryptocrack.de> 2014-01-16auth: have cgit calculate login addressJason A. Donenfeld This way we're sure to use virtual root, or any other strangeness encountered. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> 2014-01-16auth: lua string comparisons are time invariantJason A. Donenfeld By default, strings are compared by hash, so we can remove this comment. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> 2014-01-16authentication: use hidden form instead of refererJason A. Donenfeld This also gives us some CSRF protection. Note that we make use of the hmac to protect the redirect value. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> 2014-01-16auth: add basic authentication filter frameworkJason A. Donenfeld This leverages the new lua support. See filters/simple-authentication.lua for explaination of how this works. There is also additional documentation in cgitrc.5.txt. Though this is a cookie-based approach, cgit's caching mechanism is preserved for authenticated pages. Very plugable and extendable depending on user needs. The sample script uses an HMAC-SHA1 based cookie to store the currently logged in user, with an expiration date. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> 2014-01-16t0111: Additions and fixesLukas Fleischer * Rename the capitalize-* filters to dump.* since they also dump the arguments. * Add full argument validation to the email filters. Signed-off-by: Lukas Fleischer <cgit@cryptocrack.de> 2014-01-16parsing.c: Remove leading space from committerLukas Fleischer