Hello,

Anyway even if you have to use another API for large stack, it must not result in a segfault if you don't use it and have problems with the other one but in a nice handled error message telling you to use the good API or there is a flaw in Pike.

Besides you usually never know for sure you won't have large amount of svalues on the stack if there is a link with user input so I think that every program should use the macros you speak about for any user input.

BTW are you coming to the Pike camp?

/ David

Martin Stjernholm, Roxen IS @ Pike developers forum wrote:

C functions that might consume lots of stack should use check_stack to cover their needs, and maybe also check_c_stack if they recurse heavily. The main apply function checks that there are 256 svalues available on the pike stack before the call, so that "normal" functions don't need to bother with it.

C functions should never aggregate arbitrarily large amounts of svalues on the stack (e.g. to build an array). There are some helper macros to safely aggregate an array using the stack - see (BEGIN|DO|END)_AGGREGATE_ARRAY near the end of array.h.

On the pike level it's unwise to splice arbitrarily large arrays, e.g. with constructs like `+(@large_array).

/ Martin Stjernholm, Roxen IS

Previous text:

...

2004-10-13 09:25: Subject: Mysterious segfault... solved by increasing -s...

---

Hi,

Playing with sqlite module (by Bill), got some mysterious segfault after ca. 140K rows were returned... It was completely mysterious, pointing to places like:

push_text("val");	// I replaced original to test it...

where all variable values were normal (according to gdb), but valgrind went crazy... Well... The story finished when I started pike with -s1000000 - everything is OK - despite the fact that processing of 1000000 rows takes ca. 500M or RAM :)

So... The question... Isn't something that is controlled by -s (pike stack, I guess - values are accumulated on stack before aggregation) supposed to check stack size against oversizing? functions like push_* or whatever? To be honest, it took me few hours of hunting without any clue about reasons - and totally useless coredumps, valgrind reports, etc...

Ah... It was tried with pike 7.6.24... Not sure if this is relevant, but I remember that early (some time ago) I got a message like "Stack overflow" from Pike, when it was too small (even without --rtl-debug etc.).

NB: Bill, there are some troubles with sqlite3 support - I fixed them and will send a patch to you shortly... Unless you fixed those already :)

Regards, /Al

/ Brevbäraren

My new version has a proper big_query. You can look at the patched Bill-version:

/* sqlite.cmod -*- c -*- */

/* * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License as * published by the Free Software Foundation; either version 2 of the * License, or (at your option) any later version. * * This program is distributed in the hope that it will be useful, but * WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. * * $Id: sqlite.cmod,v 1.14 2004/10/07 00:33:01 nilsson Exp $ */

/* * File licensing and authorship information block. * * Version: MPL 1.1/LGPL 2.1 * * The contents of this file are subject to the Mozilla Public License Version * 1.1 (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at * http://www.mozilla.org/MPL/ * * Software distributed under the License is distributed on an "AS IS" basis, * WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License * for the specific language governing rights and limitations under the * License. * * The Initial Developer of the Original Code is * * Bill Welliver hww3@riverweb.com * * Portions created by the Initial Developer are Copyright (C) Bill Welliver * All Rights Reserved. * * Contributor(s): * * Marcus Agehall marcus.agehall@packetfront.com * Martin Nilsson nilsson@pike.ida.liu.se * * Alternatively, the contents of this file may be used under the terms of * the GNU Lesser General Public License Version 2.1 or later (the "LGPL"), * in which case the provisions of the LGPL are applicable instead * of those above. If you wish to allow use of your version of this file only * under the terms of the LGPL, and not to allow others to use your version * of this file under the terms of the MPL, indicate your decision by * deleting the provisions above and replace them with the notice * and other provisions required by the LGPL. If you do not delete * the provisions above, a recipient may use your version of this file under * the terms of any one of the MPL or the LGPL. * * Significant Contributors to this file are: * */

/*! @module Sql */

/*! @module Provider */

/*! @module SQLite */

#ifndef _GNU_SOURCE #define _GNU_SOURCE #endif

#include "util.h"

#ifdef HAVE_SQLITE #endif /* HAVE_SQLITE */

#ifdef HAVE_UNISTD_H #include <unistd.h> #endif

#include <stdlib.h> #include <stdio.h> #include <errno.h> #include <string.h>

#ifdef HAVE_SQLITE_H #include <sqlite.h> #define SQLITE_MAJOR_VERSION 2 #define SQLITE2 #else #ifdef HAVE_SQLITE3_H #include <sqlite3.h> #define SQLITE_MAJOR_VERSION 3 #define SQLITE3 #else #error No sqlite include found! #endif #endif

PIKECLASS SQLite {

#if SQLITE_MAJOR_VERSION < 3 CVAR sqlite * db; CVAR const char * errmsg; #else CVAR sqlite3 * db; #endif

#if SQLITE_MAJOR_VERSION < 3 static void clear_error_msg() { if(THIS->errmsg) { free(THIS->errmsg); THIS->errmsg = 0; } } #endif

static void handle_error(int line) { #if SQLITE_MAJOR_VERSION > 2 Pike_error("Sql.SQLite(%d): %s\n", line, sqlite3_errmsg(THIS->db)); #else if(THIS->errmsg) Pike_error("Sql.SQLite(%d): %s\n", line, THIS->errmsg); else Pike_error("Sql.SQLite(%d): Unknown error.\n"); #endif }

/**** * * Low-level SQLite interface * ****/

/*! @decl void create(string path) *! Creates a new SQLite object *! *! @param path *! sets the filename that SQLite will use as its datafile. */ PIKEFUN void create(string path, void a, void b, void c) { int res = 0; #if SQLITE_MAJOR_VERSION < 3 THIS->db = sqlite_open(path->str, 0, &THIS->errmsg); #else res = sqlite3_open(path->str, &THIS->db); #endif if(res!=SQLITE_OK || THIS->db == NULL) handle_error( __LINE__ );

pop_n_elems(args); }

PIKEFUN object big_query(string q) { int i;

i=find_identifier("query", Pike_fp->current_object->prog); apply_low(Pike_fp->current_object, i, 1);

push_constant_text( "Sql.sql_result"); SAFE_APPLY_MASTER("resolv", 1 );

stack_swap(); apply_svalue( Pike_sp-2, 1); stack_swap(); pop_stack(); }

/*! @decl int last_insert_rowid() *! Returns the last inserted rowid *! *! @returns *! the unique id of the last inserted row */ PIKEFUN int last_insert_rowid() { #if SQLITE_MAJOR_VERSION < 3 RETURN sqlite_last_insert_rowid(THIS->db); #else RETURN sqlite3_last_insert_rowid(THIS->db); #endif }

/*! @decl int changes() *! Returns the number of changed rows since the database was *! quiesced or commited. *! *! @returns *! the number of rows changed since the last commit. */ PIKEFUN int changes() { #if SQLITE_MAJOR_VERSION < 3 RETURN sqlite_changes(THIS->db); #else RETURN sqlite3_changes(THIS->db); #endif }

/*! @decl int total_changes() */ #ifdef SQLITE3 PIKEFUN int total_changes() { RETURN sqlite3_total_changes(THIS->db); } #endif

/*! @decl void interrupt() *! Interrupts the query in progress. May be called from a different *! thread or the signal handler. */ PIKEFUN void interrupt() { #if SQLITE_MAJOR_VERSION < 3 sqlite_interrupt(THIS->db); #else sqlite3_interrupt(THIS->db); #endif }

/*! @decl string version() *! *! Returns the version of the database engine. *! */ PIKEFUN string version() { push_text("SQLite " SQLITE_VERSION); }

/*! @decl array(mapping) query(string query) *! *! executes the query @[query] *! *! throws an error if the query was unsuccessful in any way. *! *! @returns *! @[1] on success if the query returns no rows (like INSERT, *! etc), or an array of mappings, one element in the array for *! each row. each row is represented by a mapping, with keys of *! the column names. */ PIKEFUN array query(string query) { struct svalue *result; #if SQLITE_MAJOR_VERSION < 3 sqlite_vm *vm; #else sqlite3_stmt *stmt; #endif

const char * tail; int r, i; int rows=0; int cols; int code;

const char ** columnName = NULL;

#if SQLITE_MAJOR_VERSION < 3 clear_error_msg(); r = sqlite_compile(THIS->db, query->str, &tail, &vm, &THIS->errmsg); #else r = sqlite3_prepare(THIS->db, query->str, query->len, &stmt, &tail); #endif

if(r != SQLITE_OK && r != SQLITE_DONE) handle_error( __LINE__ );

#if SQLITE_MAJOR_VERSION > 2 /* Cache the column names. */ cols = sqlite3_column_count(stmt); columnName = alloca(cols * sizeof(char *)); for(i=0; i<cols; i++) columnName[i] = sqlite3_column_name(stmt, i); #endif

code = 1; do { const char ** columnData;

#if SQLITE_MAJOR_VERSION < 3 r = sqlite_step(vm, &cols, &columnData, &columnName); #else r = sqlite3_step(stmt); #endif

if (r == SQLITE_ROW) { /* do something with the row */ int c = 0; rows++;

for(c=0; c<cols; c++) { #if SQLITE_MAJOR_VERSION < 3 push_text(columnName[c]); push_text(columnData[c]); #else push_text(columnName[c]); push_binary_text(sqlite3_column_blob(stmt, c), sqlite3_column_bytes(stmt, c)); #endif }

f_aggregate_mapping(cols*2); }

else if(r == SQLITE_DONE) { code=0; break; }

else if (r == SQLITE_BUSY) { /* sleep, then try again. */ #ifdef HAVE_USLEEP usleep(100); /* sleep for 100 microseconds */ #else sleep(1); #endif } else { /* some other error code was returned */ code = 0; break; } } while(code);

if(r!=SQLITE_DONE) { if(rows) pop_n_elems(rows); handle_error( __LINE__ ); } else { /* successful query retrieval */ if(rows!=0) f_aggregate(rows); }

#if SQLITE_MAJOR_VERSION < 3 r = sqlite_finalize(vm, &THIS->errmsg); #else r = sqlite3_finalize(stmt); #endif

if(r != SQLITE_OK && r != SQLITE_DONE) { if(rows) pop_stack(); handle_error( __LINE__ ); }

if(rows>0) { stack_pop_keep_top(); } else { pop_n_elems(args); f_aggregate(0); } }

INIT { THIS->db = NULL; #if SQLITE_MAJOR_VERSION < 3 THIS->errmsg = NULL; #endif }

EXIT { if(THIS->db) #if SQLITE_MAJOR_VERSION < 3 sqlite_close(THIS->db); #else sqlite3_close(THIS->db); #endif

#if SQLITE_MAJOR_VERSION < 3 if(THIS->errmsg) free(THIS->errmsg); #endif

}

} /* PIKECLASS */

/*! @endmodule */

/*! @endmodule */

/*! @endmodule */

/ Martin Nilsson (DivX Networks)

Previous text:

2004-10-13 18:15: Subject: Re: sqlite3 mod

---

On Wed, Oct 13, 2004 at 04:05:00PM +0200, Martin Nilsson (DivX Networks) @ Pike (-) developers forum wrote:

...

I've pretty much reimplemented Bills module. I'll make another iteration and check it in into Pike later. Possibly during the Pike conference.

Is there any chance to look at it? Is it usable with sqlite3? Does it support row-by-row fetching (big_query() and calling of sqlite3_step() for every fetch_row()), to avoid huge memory allocation?

Regards, /Al

/ Brevbäraren