luo980/curl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tool: add "variable" support

Browse Source 
Add support for command line variables. Set variables with --variable
name=content or --variable name@file (where "file" can be stdin if set
to a single dash (-)).

Variable content is expanded in option parameters using "{{name}}"
(without the quotes) if the option name is prefixed with
"--expand-". This gets the contents of the variable "name" inserted, or
a blank if the name does not exist as a variable. Insert "{{" verbatim
in the string by prefixing it with a backslash, like "\\{{".

Import an environment variable with --variable %name. It makes curl exit
with an error if the environment variable is not set. It can also rather
get a default value if the variable does not exist, using =content or
@file like shown above.

Example: get the USER environment variable into the URL:

 --variable %USER
 --expand-url = "https://example.com/api/{{USER}}/method"

When expanding variables, curl supports a set of functions that can make
the variable contents more convenient to use. It can trim leading and
trailing white space with "trim", output the contents as a JSON quoted
string with "json", URL encode it with "url" and base 64 encode it with
"b64". To apply functions to a variable expansion, add them colon
separated to the right side of the variable. They are then performed in
a left to right order.

Example: get the contents of a file called $HOME/.secret into a variable
called "fix". Make sure that the content is trimmed and percent-encoded
sent as POST data:

  --variable %HOME=/home/default
  --expand-variable fix@{{HOME}}/.secret
  --expand-data "{{fix:trim:url}}"
  https://example.com/

Documented. Many new test cases.

Co-brainstormed-by: Emanuele Torre
Assisted-by: Jat Satiro
Closes #11346
This commit is contained in:
Daniel Stenberg 2023-07-31 11:50:28 +02:00
parent 47a3e6e577
commit 2e160c9c65
No known key found for this signature in database
GPG Key ID: 5CC908FDB71E12C2
32 changed files with 1562 additions and 258 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
docs/cmdline-opts/Makefile.inc
View File

@ -275,6 +275,7 @@ DPAGES = \
use-ascii.d \
user-agent.d \
user.d \
variable.d \
verbose.d \
version.d \
write-out.d \

10
docs/cmdline-opts/config.d
View File

@ -21,12 +21,12 @@ if so, the colon or equals characters can be used as separators. If the option
is specified with one or two dashes, there can be no colon or equals character
between the option and its parameter.
If the parameter contains whitespace (or starts with : or =), the parameter
must be enclosed within quotes. Within double quotes, the following escape
sequences are available: \\\\, \\", \\t, \\n, \\r and \\v. A backslash
preceding any other letter is ignored.
If the parameter contains whitespace or starts with a colon (:) or equals sign
(=), it must be specified enclosed within double quotes (\&"). Within double
quotes the following escape sequences are available: \\\\, \\", \\t, \\n, \\r
and \\v. A backslash preceding any other letter is ignored.
If the first column of a config line is a '#' character, the rest of the line
If the first non-blank column of a config line is a '#' character, that line
will be treated as a comment.
Only write one option per physical line in the config file. A single line is

42
docs/cmdline-opts/page-header
View File

@ -97,6 +97,48 @@ that getting many files from the same server do not use multiple connects /
handshakes. This improves speed. Connection re-use can only be done for URLs
specified for a single command line invocation and cannot be performed between
separate curl runs.
.SH VARIABLES
Starting in curl 8.3.0, curl supports command line variables. Set variables
with --variable name=content or --variable name@file (where "file" can be
stdin if set to a single dash (-)).
Variable contents can expanded in option parameters using "{{name}}" (without
the quotes) if the option name is prefixed with "--expand-". This gets the
contents of the variable "name" inserted, or a blank if the name does not
exist as a variable. Insert "{{" verbatim in the string by prefixing it with a
backslash, like "\\{{".
You an access and expand environment variables by first importing them. You
can select to either require the environment variable to be set or you can
provide a default value in case it is not already set. Plain --variable %name
imports the variable called 'name' but exits with an error if that environment
variable is not alreadty set. To provide a default value if it is not set, use
--variable %name=content or --variable %name@content.
Example. Get the USER environment variable into the URL, fail if USER is not
set:
--variable '%USER'
--expand-url = "https://example.com/api/{{USER}}/method"
When expanding variables, curl supports a set of functions that can make the
variable contents more convenient to use. It can trim leading and trailing
white space with "trim", it can output the contents as a JSON quoted string
with "json" and it can URL encode the string with "urlencode". You apply
function to a variable expansion, add them colon separated to the right side
of the variable. Variable content holding null bytes that are not encoded when
expanded, will cause error.
Exmaple: get the contents of a file called $HOME/.secret into a variable
called "fix". Make sure that the content is trimmed and percent-encoded sent
as POST data:
--variable %HOME
--expand-variable fix@{{HOME}}/.secret
--expand-data "{{fix:trim:urlencode}}"
https://example.com/
Command line variables and expansions were added in in 8.3.0.
.SH OUTPUT
If not told otherwise, curl writes the received data to stdout. It can be
instructed to instead save that data into a local file, using the --output or

50
docs/cmdline-opts/variable.d Normal file
View File

@ -0,0 +1,50 @@
c: Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
SPDX-License-Identifier: curl
Long: variable
Arg: <[%]name=text/@file>
Help: Set variable
Category: curl
Example: --variable name=smith $URL
Added: 8.3.0
See-also: config
Multi: append
---
Set a variable with "name=content" or "name@file" (where "file" can be stdin
if set to a single dash (-)). The name is a case sensitive identifier that
must consist of no other letters than a-z, A-Z, 0-9 or underscore. The
specified content is then associated with this identifier.
The name must be unique within a command line invoke, setting the same
variable name again will be ignored.
The contents of a variable can be referenced in a later command line option
when that option name is prefixed with "--expand-", and the name is used as
"{{name}}" (without the quotes).
--variable can import environment variables into the name space. Opt to either
require the environment variable to be set or provide a default value for the
variable in case it is not already set.
--variable %name imports the variable called 'name' but exits with an error if
that environment variable is not alreadty set. To provide a default value if
the environment variable is not set, use --variable %name=content or
--variable %name@content. Note that on some systems - but not all -
environment variables are case insensitive.
When expanding variables, curl supports a set of functions that can make the
variable contents more convenient to use. You apply a function to a variable
expansion by adding a colon and then list the desired functions in a
comma-separted list that is evaluated in a left-to-right order. Variable
content holding null bytes that are not encoded when expanded, will cause
error.
These are functions that can help you get the value inserted more
conveniently.
"trim" removes all leading and trailing white space.
"json" outputs the content using JSON string quoting rules.
"url" shows the content URL (percent) encoded.
"b64" expands the variable base64 encoded

1
docs/options-in-versions
View File

@ -261,6 +261,7 @@
--use-ascii (-B) 5.0
--user (-u) 4.0
--user-agent (-A) 4.5.1
--variable 8.3.0
--verbose (-v) 4.0
--version (-V) 4.0
--write-out (-w) 6.5

5
lib/base64.c
View File

@ -32,13 +32,14 @@
!defined(CURL_DISABLE_POP3) || \
!defined(CURL_DISABLE_IMAP) || \
!defined(CURL_DISABLE_DOH) || defined(USE_SSL)
#include "urldata.h" /* for the Curl_easy definition */
#include "curl/curl.h"
#include "warnless.h"
#include "curl_base64.h"
/* The last 2 #include files should be in this order */
#ifdef BUILDING_LIBCURL
#include "curl_memory.h"
#endif
#include "memdebug.h"
/* ---- Base64 Encoding/Decoding Table --- */

9
lib/curl_base64.h
View File

@ -24,11 +24,18 @@
*
***************************************************************************/
#ifndef BUILDING_LIBCURL
/* this renames functions so that the tool code can use the same code
without getting symbol collisions */
#define Curl_base64_encode(a,b,c,d) curlx_base64_encode(a,b,c,d)
#define Curl_base64url_encode(a,b,c,d) curlx_base64url_encode(a,b,c,d)
#define Curl_base64_decode(a,b,c) curlx_base64_decode(a,b,c)
#endif
CURLcode Curl_base64_encode(const char *inputbuff, size_t insize,
char **outptr, size_t *outlen);
CURLcode Curl_base64url_encode(const char *inputbuff, size_t insize,
char **outptr, size_t *outlen);
CURLcode Curl_base64_decode(const char *src,
unsigned char **outptr, size_t *outlen);
#endif /* HEADER_CURL_BASE64_H */

2
projects/generate.bat
View File

@ -217,6 +217,7 @@ rem
call :element %1 lib "curl_multibyte.c" %3
call :element %1 lib "version_win32.c" %3
call :element %1 lib "dynbuf.c" %3
call :element %1 lib "base64.c" %3
) else if "!var!" == "CURL_SRC_X_H_FILES" (
call :element %1 lib "config-win32.h" %3
call :element %1 lib "curl_setup.h" %3
@ -228,6 +229,7 @@ rem
call :element %1 lib "curl_multibyte.h" %3
call :element %1 lib "version_win32.h" %3
call :element %1 lib "dynbuf.h" %3
call :element %1 lib "curl_base64.h" %3
) else if "!var!" == "CURL_LIB_C_FILES" (
for /f "delims=" %%c in ('dir /b ..\lib\*.c') do call :element %1 lib "%%c" %3
) else if "!var!" == "CURL_LIB_H_FILES" (

2
src/CMakeLists.txt
View File

@ -63,7 +63,7 @@ endif()
# CURL_CFILES, CURLX_CFILES, CURL_HFILES come from Makefile.inc
if(BUILD_STATIC_CURL)
set(CURLX_CFILES ../lib/dynbuf.c)
set(CURLX_CFILES ../lib/dynbuf.c ../lib/base64.c)
endif()
add_executable(

15
src/Makefile.inc
View File

@ -32,13 +32,14 @@
# libcurl has sources that provide functions named curlx_* that aren't part of
# the official API, but we re-use the code here to avoid duplication.
CURLX_CFILES = \
../lib/base64.c \
../lib/curl_multibyte.c \
../lib/dynbuf.c \
../lib/nonblock.c \
../lib/strtoofft.c \
../lib/timediff.c \
../lib/nonblock.c \
../lib/warnless.c \
../lib/curl_multibyte.c \
../lib/version_win32.c \
../lib/dynbuf.c
../lib/warnless.c
CURLX_HFILES = \
../lib/curl_setup.h \
@ -91,7 +92,8 @@ CURL_CFILES = \
tool_vms.c \
tool_writeout.c \
tool_writeout_json.c \
tool_xattr.c
tool_xattr.c \
var.c
CURL_HFILES = \
slist_wc.h \
@ -135,7 +137,8 @@ CURL_HFILES = \
tool_vms.h \
tool_writeout.h \
tool_writeout_json.h \
tool_xattr.h
tool_xattr.h \
var.h
CURL_RCFILES = curl.rc

2
src/tool_cfgable.h
View File

@ -26,6 +26,7 @@
#include "tool_setup.h"
#include "tool_sdecls.h"
#include "tool_urlglob.h"
#include "var.h"
struct GlobalConfig;
@ -322,6 +323,7 @@ struct GlobalConfig {
unsigned short parallel_max; /* MAX_PARALLEL is the maximum */
bool parallel_connect;
char *help_category; /* The help category, if set */
struct var *variables;
struct OperationConfig *first;
struct OperationConfig *current;
struct OperationConfig *last; /* Always last in the struct */

497
src/tool_getparam.c
View File

File diff suppressed because it is too large Load Diff

1
src/tool_getparam.h
View File

@ -48,6 +48,7 @@ typedef enum {
PARAM_CONTDISP_SHOW_HEADER, /* --include and --remote-header-name */
PARAM_CONTDISP_RESUME_FROM, /* --continue-at and --remote-header-name */
PARAM_READ_ERROR,
PARAM_EXPAND_ERROR, /* --expand problem */
PARAM_LAST
} ParameterError;

2
src/tool_helpers.c
View File

@ -76,6 +76,8 @@ const char *param2text(int res)
return "--continue-at and --remote-header-name cannot be combined";
case PARAM_READ_ERROR:
return "error encountered when reading a file";
case PARAM_EXPAND_ERROR:
return "variable expansion failure";
default:
return "unknown error";
}

11
src/tool_listhelp.c
View File

@ -246,12 +246,12 @@ const struct helptxt helptext[] = {
{" --happy-eyeballs-timeout-ms <milliseconds>",
"Time for IPv6 before trying IPv4",
CURLHELP_CONNECTION},
{" --haproxy-clientip",
"Sets client IP in HAProxy PROXY protocol v1 header",
CURLHELP_HTTP | CURLHELP_PROXY},
{" --haproxy-protocol",
"Send HAProxy PROXY protocol v1 header",
CURLHELP_HTTP | CURLHELP_PROXY},
{" --haproxy-clientip",
"Sets the HAProxy PROXY protocol v1 client IP",
CURLHELP_HTTP | CURLHELP_PROXY},
{"-I, --head",
"Show document info only",
CURLHELP_HTTP | CURLHELP_FTP | CURLHELP_FILE},
@ -760,7 +760,7 @@ const struct helptxt helptext[] = {
"Like --trace, but without hex output",
CURLHELP_VERBOSE},
{" --trace-ids",
"Add transfer/connection identifiers to trace/verbose output",
"Add transfer and connection identifiers to trace/verbose output",
CURLHELP_VERBOSE},
{" --trace-time",
"Add time stamps to trace/verbose output",
@ -786,6 +786,9 @@ const struct helptxt helptext[] = {
{"-A, --user-agent <name>",
"Send User-Agent <name> to server",
CURLHELP_IMPORTANT | CURLHELP_HTTP},
{" --variable <name=text/@file>",
"Set variable",
CURLHELP_CURL},
{"-v, --verbose",
"Make the operation more talkative",
CURLHELP_IMPORTANT | CURLHELP_VERBOSE},

49
src/tool_operate.c
View File

@ -2756,36 +2756,39 @@ CURLcode operate(struct GlobalConfig *global, int argc, argv_item_t argv[])
/* Cleanup the libcurl source output */
easysrc_cleanup();
}
return CURLE_OUT_OF_MEMORY;
result = CURLE_OUT_OF_MEMORY;
}
curl_share_setopt(share, CURLSHOPT_SHARE, CURL_LOCK_DATA_COOKIE);
curl_share_setopt(share, CURLSHOPT_SHARE, CURL_LOCK_DATA_DNS);
curl_share_setopt(share, CURLSHOPT_SHARE, CURL_LOCK_DATA_SSL_SESSION);
curl_share_setopt(share, CURLSHOPT_SHARE, CURL_LOCK_DATA_CONNECT);
curl_share_setopt(share, CURLSHOPT_SHARE, CURL_LOCK_DATA_PSL);
curl_share_setopt(share, CURLSHOPT_SHARE, CURL_LOCK_DATA_HSTS);
if(!result) {
curl_share_setopt(share, CURLSHOPT_SHARE, CURL_LOCK_DATA_COOKIE);
curl_share_setopt(share, CURLSHOPT_SHARE, CURL_LOCK_DATA_DNS);
curl_share_setopt(share, CURLSHOPT_SHARE,
CURL_LOCK_DATA_SSL_SESSION);
curl_share_setopt(share, CURLSHOPT_SHARE, CURL_LOCK_DATA_CONNECT);
curl_share_setopt(share, CURLSHOPT_SHARE, CURL_LOCK_DATA_PSL);
curl_share_setopt(share, CURLSHOPT_SHARE, CURL_LOCK_DATA_HSTS);
/* Get the required arguments for each operation */
do {
result = get_args(operation, count++);
/* Get the required arguments for each operation */
do {
result = get_args(operation, count++);
operation = operation->next;
} while(!result && operation);
operation = operation->next;
} while(!result && operation);
/* Set the current operation pointer */
global->current = global->first;
/* Set the current operation pointer */
global->current = global->first;
/* now run! */
result = run_all_transfers(global, share, result);
/* now run! */
result = run_all_transfers(global, share, result);
curl_share_cleanup(share);
if(global->libcurl) {
/* Cleanup the libcurl source output */
easysrc_cleanup();
curl_share_cleanup(share);
if(global->libcurl) {
/* Cleanup the libcurl source output */
easysrc_cleanup();
/* Dump the libcurl code if previously enabled */
dumpeasysrc(global);
/* Dump the libcurl code if previously enabled */
dumpeasysrc(global);
}
}
}
else
@ -2793,5 +2796,7 @@ CURLcode operate(struct GlobalConfig *global, int argc, argv_item_t argv[])
}
}
varcleanup(global);
return result;
}

61
src/tool_writeout_json.c
View File

@ -31,50 +31,73 @@
#include "tool_writeout_json.h"
#include "tool_writeout.h"
void jsonWriteString(FILE *stream, const char *in, bool lowercase)
#define MAX_JSON_STRING 100000
/* provide the given string in dynbuf as a quoted json string, but without the
outer quotes. The buffer is not inited by this function.
Return 0 on success, non-zero on error.
*/
int jsonquoted(const char *in, size_t len,
struct curlx_dynbuf *out, bool lowercase)
{
const char *i = in;
const char *in_end = in + strlen(in);
const char *in_end = &in[len];
CURLcode result = CURLE_OK;
fputc('\"', stream);
for(; i < in_end; i++) {
for(; (i < in_end) && !result; i++) {
switch(*i) {
case '\\':
fputs("\\\\", stream);
result = curlx_dyn_addn(out, "\\\\", 2);
break;
case '\"':
fputs("\\\"", stream);
result = curlx_dyn_addn(out, "\\\"", 2);
break;
case '\b':
fputs("\\b", stream);
result = curlx_dyn_addn(out, "\\b", 2);
break;
case '\f':
fputs("\\f", stream);
result = curlx_dyn_addn(out, "\\f", 2);
break;
case '\n':
fputs("\\n", stream);
result = curlx_dyn_addn(out, "\\n", 2);
break;
case '\r':
fputs("\\r", stream);
result = curlx_dyn_addn(out, "\\r", 2);
break;
case '\t':
fputs("\\t", stream);
result = curlx_dyn_addn(out, "\\t", 2);
break;
default:
if(*i < 32) {
fprintf(stream, "\\u%04x", *i);
}
if(*i < 32)
result = curlx_dyn_addf(out, "\\u%04x", *i);
else {
char out = *i;
if(lowercase && (out >= 'A' && out <= 'Z'))
char o = *i;
if(lowercase && (o >= 'A' && o <= 'Z'))
/* do not use tolower() since that's locale specific */
out |= ('a' - 'A');
fputc(out, stream);
o |= ('a' - 'A');
result = curlx_dyn_addn(out, &o, 1);
}
break;
}
}
fputc('\"', stream);
if(result)
return (int)result;
return 0;
}
void jsonWriteString(FILE *stream, const char *in, bool lowercase)
{
struct curlx_dynbuf out;
curlx_dyn_init(&out, MAX_JSON_STRING);
if(!jsonquoted(in, strlen(in), &out, lowercase)) {
fputc('\"', stream);
if(curlx_dyn_len(&out))
fputs(curlx_dyn_ptr(&out), stream);
fputc('\"', stream);
}
curlx_dyn_free(&out);
}
void ourWriteOutJSON(FILE *stream, const struct writeoutvar mappings[],

3
src/tool_writeout_json.h
View File

@ -26,6 +26,9 @@
#include "tool_setup.h"
#include "tool_writeout.h"
int jsonquoted(const char *in, size_t len,
struct curlx_dynbuf *out, bool lowercase);
void ourWriteOutJSON(FILE *stream, const struct writeoutvar mappings[],
struct per_transfer *per, CURLcode per_result);
void headerJSON(FILE *stream, struct per_transfer *per);

462
src/var.c Normal file
View File

@ -0,0 +1,462 @@
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
* SPDX-License-Identifier: curl
*
***************************************************************************/
#include "tool_setup.h"
#define ENABLE_CURLX_PRINTF
/* use our own printf() functions */
#include "curlx.h"
#include "tool_cfgable.h"
#include "tool_getparam.h"
#include "tool_helpers.h"
#include "tool_findfile.h"
#include "tool_msgs.h"
#include "tool_parsecfg.h"
#include "dynbuf.h"
#include "curl_base64.h"
#include "tool_paramhlp.h"
#include "tool_writeout_json.h"
#include "var.h"
#include "memdebug.h" /* keep this as LAST include */
#define MAX_EXPAND_CONTENT 10000000
static char *Memdup(const char *data, size_t len)
{
char *p = malloc(len + 1);
if(!p)
return NULL;
if(len)
memcpy(p, data, len);
p[len] = 0;
return p;
}
/* free everything */
void varcleanup(struct GlobalConfig *global)
{
struct var *list = global->variables;
while(list) {
struct var *t = list;
list = list->next;
free((char *)t->content);
free((char *)t->name);
free(t);
}
}
static const struct var *varcontent(struct GlobalConfig *global,
const char *name, size_t nlen)
{
struct var *list = global->variables;
while(list) {
if((strlen(list->name) == nlen) &&
!strncmp(name, list->name, nlen)) {
return list;
}
list = list->next;
}
return NULL;
}
#define ENDOFFUNC(x) (((x) == '}') || ((x) == ':'))
#define FUNCMATCH(ptr,name,len) \
(!strncmp(ptr, name, len) && ENDOFFUNC(ptr[len]))
#define FUNC_TRIM "trim"
#define FUNC_TRIM_LEN (sizeof(FUNC_TRIM) - 1)
#define FUNC_JSON "json"
#define FUNC_JSON_LEN (sizeof(FUNC_JSON) - 1)
#define FUNC_URL "url"
#define FUNC_URL_LEN (sizeof(FUNC_URL) - 1)
#define FUNC_B64 "b64"
#define FUNC_B64_LEN (sizeof(FUNC_B64) - 1)
static ParameterError varfunc(struct GlobalConfig *global,
char *c, /* content */
size_t clen, /* content length */
char *f, /* functions */
size_t flen, /* function string length */
struct curlx_dynbuf *out)
{
bool alloc = FALSE;
ParameterError err = PARAM_OK;
const char *finput = f;
/* The functions are independent and runs left to right */
while(*f && !err) {
if(*f == '}')
/* end of functions */
break;
/* On entry, this is known to be a colon already. In subsequent laps, it
is also known to be a colon since that is part of the FUNCMATCH()
checks */
f++;
if(FUNCMATCH(f, FUNC_TRIM, FUNC_TRIM_LEN)) {
size_t len = clen;
f += FUNC_TRIM_LEN;
if(clen) {
/* skip leading white space, including CRLF */
while(*c && ISSPACE(*c)) {
c++;
len--;
}
while(len && ISSPACE(c[len-1]))
len--;
}
/* put it in the output */
curlx_dyn_reset(out);
if(curlx_dyn_addn(out, c, len)) {
err = PARAM_NO_MEM;
break;
}
}
else if(FUNCMATCH(f, FUNC_JSON, FUNC_JSON_LEN)) {
f += FUNC_JSON_LEN;
curlx_dyn_reset(out);
if(clen) {
if(jsonquoted(c, clen, out, FALSE)) {
err = PARAM_NO_MEM;
break;
}
}
}
else if(FUNCMATCH(f, FUNC_URL, FUNC_URL_LEN)) {
f += FUNC_URL_LEN;
curlx_dyn_reset(out);
if(clen) {
char *enc = curl_easy_escape(NULL, c, (int)clen);
if(!enc) {
err = PARAM_NO_MEM;
break;
}
/* put it in the output */
if(curlx_dyn_add(out, enc)) {
err = PARAM_NO_MEM;
break;
}
curl_free(enc);
}
}
else if(FUNCMATCH(f, FUNC_B64, FUNC_B64_LEN)) {
f += FUNC_B64_LEN;
curlx_dyn_reset(out);
if(clen) {
char *enc;
size_t elen;
CURLcode result = curlx_base64_encode(c, clen, &enc, &elen);
if(result) {
err = PARAM_NO_MEM;
break;
}
/* put it in the output */
if(curlx_dyn_addn(out, enc, elen))
err = PARAM_NO_MEM;
curl_free(enc);
if(err)
break;
}
}
else {
/* unsupported function */
errorf(global, "unknown variable function in '%.*s'",
(int)flen, finput);
err = PARAM_EXPAND_ERROR;
break;
}
if(alloc)
free(c);
clen = curlx_dyn_len(out);
c = Memdup(curlx_dyn_ptr(out), clen);
if(!c) {
err = PARAM_NO_MEM;
break;
}
alloc = TRUE;
}
if(alloc)
free(c);
if(err)
curlx_dyn_free(out);
return err;
}
ParameterError varexpand(struct GlobalConfig *global,
const char *line, struct curlx_dynbuf *out,
bool *replaced)
{
CURLcode result;
char *envp;
bool added = FALSE;
const char *input = line;
*replaced = FALSE;
curlx_dyn_init(out, MAX_EXPAND_CONTENT);
do {
envp = strstr(line, "{{");
if((envp > line) && envp[-1] == '\\') {
/* preceding backslash, we want this verbatim */
/* insert the text up to this point, minus the backslash */
result = curlx_dyn_addn(out, line, envp - line - 1);
if(result)
return PARAM_NO_MEM;
/* output '{{' then continue from here */
result = curlx_dyn_addn(out, "{{", 2);
if(result)
return PARAM_NO_MEM;
line = &envp[2];
}
else if(envp) {
char name[128];
size_t nlen;
size_t i;
char *funcp;
char *clp = strstr(envp, "}}");
size_t prefix;
if(!clp) {
/* uneven braces */
warnf(global, "missing close '}}' in '%s'", input);
break;
}
prefix = 2;
envp += 2; /* move over the {{ */
/* if there is a function, it ends the name with a colon */
funcp = memchr(envp, ':', clp - envp);
if(funcp)
nlen = funcp - envp;
else
nlen = clp - envp;
if(!nlen || (nlen >= sizeof(name))) {
warnf(global, "bad variable name length '%s'", input);
/* insert the text as-is since this is not an env variable */
result = curlx_dyn_addn(out, line, clp - line + prefix);
if(result)
return PARAM_NO_MEM;
}
else {
/* insert the text up to this point */
result = curlx_dyn_addn(out, line, envp - prefix - line);
if(result)
return PARAM_NO_MEM;
/* copy the name to separate buffer */
memcpy(name, envp, nlen);
name[nlen] = 0;
/* verify that the name looks sensible */
for(i = 0; (i < nlen) &&
(ISALNUM(name[i]) || (name[i] == '_')); i++);
if(i != nlen) {
warnf(global, "bad variable name: %s", name);
/* insert the text as-is since this is not an env variable */
result = curlx_dyn_addn(out, envp - prefix,
clp - envp + prefix + 2);
if(result)
return PARAM_NO_MEM;
}
else {
char *value;
size_t vlen = 0;
struct curlx_dynbuf buf;
const struct var *v = varcontent(global, name, nlen);
if(v) {
value = (char *)v->content;
vlen = v->clen;
}
else
value = NULL;
curlx_dyn_init(&buf, MAX_EXPAND_CONTENT);
if(funcp) {
/* apply the list of functions on the value */
size_t flen = clp - funcp;
ParameterError err = varfunc(global, value, vlen, funcp, flen,
&buf);
if(err)
return err;
value = curlx_dyn_ptr(&buf);
vlen = curlx_dyn_len(&buf);
}
if(value && *value) {
/* A variable might contain null bytes. Such bytes cannot be shown
using normal means, this is an error. */
char *nb = memchr(value, '\0', vlen);
if(nb) {
errorf(global, "variable contains null byte");
return PARAM_EXPAND_ERROR;
}
}
/* insert the value */
result = curlx_dyn_addn(out, value, vlen);
curlx_dyn_free(&buf);
if(result)
return PARAM_NO_MEM;
added = true;
}
}
line = &clp[2];
}
} while(envp);
if(added && *line) {
/* add the "suffix" as well */
result = curlx_dyn_add(out, line);
if(result)
return PARAM_NO_MEM;
}
*replaced = added;
if(!added)
curlx_dyn_free(out);
return PARAM_OK;
}
/*
* Created in a way that is not revealing how variables is actually stored so
* that we can improve this if we want better performance when managing many
* at a later point.
*/
static ParameterError addvariable(struct GlobalConfig *global,
const char *name,
size_t nlen,
const char *content,
size_t clen,
bool contalloc)
{
struct var *p;
const struct var *check = varcontent(global, name, nlen);
if(check)
notef(global, "Overwriting variable '%s'", check->name);
p = calloc(sizeof(struct var), 1);
if(!p)
return PARAM_NO_MEM;
p->name = Memdup(name, nlen);
if(!p->name)
goto err;
p->content = contalloc ? content: Memdup(content, clen);
if(!p->content)
goto err;
p->clen = clen;
p->next = global->variables;
global->variables = p;
return PARAM_OK;
err:
free((char *)p->content);
free((char *)p->name);
free(p);
return PARAM_NO_MEM;
}
ParameterError setvariable(struct GlobalConfig *global,
const char *input)
{
const char *name;
size_t nlen;
char *content = NULL;
size_t clen = 0;
bool contalloc = FALSE;
const char *line = input;
ParameterError err = PARAM_OK;
bool import = FALSE;
char *ge = NULL;
if(*input == '%') {
import = TRUE;
line++;
}
name = line;
while(*line && (ISALNUM(*line) || (*line == '_')))
line++;
nlen = line - name;
if(!nlen || (nlen > 128)) {
warnf(global, "Bad variable name length (%zd), skipping", nlen);
return PARAM_OK;
}
if(import) {
ge = curl_getenv(name);
if(!*line && !ge) {
/* no assign, no variable, fail */
errorf(global, "Variable '%s' import fail, not set", name);
return PARAM_EXPAND_ERROR;
}
else if(ge) {
/* there is a value to use */
content = ge;
clen = strlen(ge);
}
}
if(content)
;
else if(*line == '@') {
/* read from file or stdin */
FILE *file;
bool use_stdin;
line++;
use_stdin = !strcmp(line, "-");
if(use_stdin)
file = stdin;
else {
file = fopen(line, "rb");
}
if(file) {
err = file2memory(&content, &clen, file);
/* in case of out of memory, this should fail the entire operation */
contalloc = TRUE;
}
if(!use_stdin)
fclose(file);
if(err)
return err;
}
else if(*line == '=') {
line++;
/* this is the exact content */
content = (char *)line;
clen = strlen(line);
}
else {
warnf(global, "Bad --variable syntax, skipping: %s", input);
return PARAM_OK;
}
err = addvariable(global, name, nlen, content, clen, contalloc);
if(err) {
if(contalloc)
free(content);
}
curl_free(ge);
return err;
}

48
src/var.h Normal file
View File

@ -0,0 +1,48 @@
#ifndef HEADER_CURL_VAR_H
#define HEADER_CURL_VAR_H
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
* SPDX-License-Identifier: curl
*
***************************************************************************/
#include "tool_getparam.h"
#include "dynbuf.h"
struct var {
struct var *next;
const char *name;
const char *content;
size_t clen; /* content length */
};
struct GlobalConfig;
ParameterError setvariable(struct GlobalConfig *global, const char *input);
ParameterError varexpand(struct GlobalConfig *global,
const char *line, struct curlx_dynbuf *out,
bool *replaced);
/* free everything */
void varcleanup(struct GlobalConfig *global);
#endif /* HEADER_CURL_VAR_H */

7
tests/data/Makefile.inc
View File

@ -69,10 +69,11 @@ test390 test391 test392 test393 test394 test395 test396 test397 test398 \
test399 test400 test401 test402 test403 test404 test405 test406 test407 \
test408 test409 test410 test411 test412 test413 test414 test415 test416 \
test417 test418 test419 test420 test421 test422 test423 test424 test425 \
test426 test427 \
test430 test431 test432 test433 test434 test435 test436 \
test426 test427 test428 test429 test430 test431 test432 test433 test434 \
test435 test436 \
\
test440 test441 test442 test443 test444 test445 test446 test447 \
test440 test441 test442 test443 test444 test445 test446 test447 test448 \
test449 test450 test451 test452 test453 test454 test455 \
\
test490 test491 test492 test493 test494 test495 test496 \
\

68
tests/data/test428 Normal file
View File

@ -0,0 +1,68 @@
<testcase>
<info>
<keywords>
HTTP
variables
</keywords>
</info>
#
# Server-side
<reply>
<data crlf="yes">
HTTP/1.1 200 OK
Date: Tue, 09 Nov 2010 14:49:00 GMT
Server: test-server/fake
Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
ETag: "21025-dc7-39462498"
Accept-Ranges: bytes
Content-Length: 6
Connection: close
Content-Type: text/html
Funny-head: yesyes
-foo-
</data>
</reply>
#
# Client-side
<client>
<server>
http
</server>
<setenv>
FUNVALUE=contents
VALUE2=curl
BLANK=
</setenv>
<name>
Expand environment variables within config file
</name>
<file name="%LOGDIR/cmd">
--variable %FUNVALUE
--variable %VALUE2
--variable %BLANK
--variable %curl_NOT_SET=default
--expand-data 1{{FUNVALUE}}2{{VALUE2}}3{{curl_NOT_SET}}4{{BLANK}}5\{{verbatim}}6{{not.good}}7{{}}
</file>
<command>
http://%HOSTIP:%HTTPPORT/%TESTNUMBER -K %LOGDIR/cmd
</command>
</client>
#
# Verify data after the test has been "shot"
<verify>
<protocol crlf="yes" nonewline="yes">
POST /%TESTNUMBER HTTP/1.1
Host: %HOSTIP:%HTTPPORT
User-Agent: curl/%VERSION
Accept: */*
Content-Length: 54
Content-Type: application/x-www-form-urlencoded
1contents2curl3default45{{verbatim}}6{{not.good}}7{{}}
</protocol>
</verify>
</testcase>

63
tests/data/test429 Normal file
View File

@ -0,0 +1,63 @@
<testcase>
<info>
<keywords>
HTTP
HTTP POST
variables
</keywords>
</info>
#
# Server-side
<reply>
<data crlf="yes">
HTTP/1.1 200 OK
Date: Tue, 09 Nov 2010 14:49:00 GMT
Server: test-server/fake
Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
ETag: "21025-dc7-39462498"
Accept-Ranges: bytes
Content-Length: 6
Connection: close
Content-Type: text/html
Funny-head: yesyes
-foo-
</data>
</reply>
#
# Client-side
<client>
<server>
http
</server>
<setenv>
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF=contents2023
</setenv>
<name>
Expand environment variable in config file - too long name
</name>
<file name="%LOGDIR/cmd">
--expand-data {{FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF}}
</file>
<command>
http://%HOSTIP:%HTTPPORT/%TESTNUMBER -K %LOGDIR/cmd
</command>
</client>
#
# Verify data after the test has been "shot"
<verify>
<protocol crlf="yes" nonewline="yes">
POST /%TESTNUMBER HTTP/1.1
Host: %HOSTIP:%HTTPPORT
User-Agent: curl/%VERSION
Accept: */*
Content-Length: 133
Content-Type: application/x-www-form-urlencoded
{{FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF}}
</protocol>
</verify>
</testcase>

67
tests/data/test448 Normal file
View File

@ -0,0 +1,67 @@
<testcase>
<info>
<keywords>
HTTP
variables
--config
</keywords>
</info>
#
# Server-side
<reply>
<data crlf="yes">
HTTP/1.1 200 OK
Date: Tue, 09 Nov 2010 14:49:00 GMT
Server: test-server/fake
Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
ETag: "21025-dc7-39462498"
Accept-Ranges: bytes
Content-Length: 6
Connection: close
Content-Type: text/html
Funny-head: yesyes
-foo-
</data>
</reply>
#
# Client-side
<client>
<server>
http
</server>
<setenv>
FUNVALUE=contents
VALUE2=curl
BLANK=
</setenv>
<name>
Environment variables within config file, unbalanced braces
</name>
<file name="%LOGDIR/cmd">
--variable %FUNVALUE
--variable %VALUE2
--expand-data 1{{FUNVALUE}}2{{VALUE2}}3{{curl_NOT_SET}}4{{AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA}}5{{broken
</file>
<command>
http://%HOSTIP:%HTTPPORT/%TESTNUMBER -K %LOGDIR/cmd
</command>
</client>
#
# Verify data after the test has been "shot"
<verify>
<protocol crlf="yes" nonewline="yes">
POST /%TESTNUMBER HTTP/1.1
Host: %HOSTIP:%HTTPPORT
User-Agent: curl/%VERSION
Accept: */*
Content-Length: 157
Content-Type: application/x-www-form-urlencoded
1contents2curl34{{AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA}}5{{broken
</protocol>
</verify>
</testcase>

65
tests/data/test449 Normal file
View File

@ -0,0 +1,65 @@
<testcase>
<info>
<keywords>
HTTP
variables
--config
</keywords>
</info>
#
# Server-side
<reply>
<data crlf="yes">
HTTP/1.1 200 OK
Date: Tue, 09 Nov 2010 14:49:00 GMT
Server: test-server/fake
Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
ETag: "21025-dc7-39462498"
Accept-Ranges: bytes
Content-Length: 6
Connection: close
Content-Type: text/html
Funny-head: yesyes
-foo-
</data>
</reply>
#
# Client-side
<client>
<server>
http
</server>
<setenv>
FUNVALUE=contents
VALUE2=curl
BLANK=
</setenv>
<name>
Environment variables in config file w/o [expand]
</name>
<file name="%LOGDIR/cmd">
-d 1{{FUNVALUE}}2{{VALUE2}}3{{CURL_NOT_SET}}4{{BLANK}}5\{{verbatim}}6{{not.good}}7{{}}
</file>
<command>
http://%HOSTIP:%HTTPPORT/%TESTNUMBER -K %LOGDIR/cmd
</command>
</client>
#
# Verify data after the test has been "shot"
<verify>
<protocol crlf="yes" nonewline="yes">
POST /%TESTNUMBER HTTP/1.1
Host: %HOSTIP:%HTTPPORT
User-Agent: curl/%VERSION
Accept: */*
Content-Length: 83
Content-Type: application/x-www-form-urlencoded
1{{FUNVALUE}}2{{VALUE2}}3{{CURL_NOT_SET}}4{{BLANK}}5\{{verbatim}}6{{not.good}}7{{}}
</protocol>
</verify>
</testcase>

60
tests/data/test450 Normal file
View File

@ -0,0 +1,60 @@
<testcase>
<info>
<keywords>
HTTP
--config
variables
</keywords>
</info>
#
# Server-side
<reply>
<data crlf="yes">
HTTP/1.1 200 OK
Date: Tue, 09 Nov 2010 14:49:00 GMT
Server: test-server/fake
Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
ETag: "21025-dc7-39462498"
Accept-Ranges: bytes
Content-Length: 6
Connection: close
Content-Type: text/html
Funny-head: yesyes
-foo-
</data>
</reply>
#
# Client-side
<client>
<server>
http
</server>
<name>
Variable from file that is trimmed and URL encoded
</name>
<file name="%LOGDIR/junk">
space with space
</file>
<command>
http://%HOSTIP:%HTTPPORT/%TESTNUMBER --variable what@%LOGDIR/junk --expand-data "{{what:trim:url}}"
</command>
</client>
#
# Verify data after the test has been "shot"
<verify>
<protocol crlf="yes" nonewline="yes">
POST /%TESTNUMBER HTTP/1.1
Host: %HOSTIP:%HTTPPORT
User-Agent: curl/%VERSION
Accept: */*
Content-Length: 20
Content-Type: application/x-www-form-urlencoded
space%20with%20space
</protocol>
</verify>
</testcase>

59
tests/data/test451 Normal file
View File

@ -0,0 +1,59 @@
<testcase>
<info>
<keywords>
HTTP
variables
</keywords>
</info>
#
# Server-side
<reply>
<data crlf="yes">
HTTP/1.1 200 OK
Date: Tue, 09 Nov 2010 14:49:00 GMT
Server: test-server/fake
Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
ETag: "21025-dc7-39462498"
Accept-Ranges: bytes
Content-Length: 6
Connection: close
Content-Type: text/html
Funny-head: yesyes
-foo-
</data>
</reply>
#
# Client-side
<client>
<server>
http
</server>
<name>
Variable from file that is JSON and URL encoded (with null byte)
</name>
<file name="%LOGDIR/junk">
%hex[%01%02%03%00%04%05%06]hex%
</file>
<command>
http://%HOSTIP:%HTTPPORT/%TESTNUMBER --variable what@%LOGDIR/junk --variable second=hello --variable second=again --expand-data "--{{what:trim:json}}22{{none}}--{{second}}{{what:trim:url}}"
</command>
</client>
#
# Verify data after the test has been "shot"
<verify>
<protocol crlf="yes" nonewline="yes">
POST /%TESTNUMBER HTTP/1.1
Host: %HOSTIP:%HTTPPORT
User-Agent: curl/%VERSION
Accept: */*
Content-Length: 74
Content-Type: application/x-www-form-urlencoded
--\u0001\u0002\u0003\u0000\u0004\u0005\u000622--again%01%02%03%00%04%05%06
</protocol>
</verify>
</testcase>

34
tests/data/test452 Normal file
View File

@ -0,0 +1,34 @@
<testcase>
<info>
<keywords>
variables
</keywords>
</info>
#
# Server-side
<reply>
</reply>
#
# Client-side
<client>
<server>
http
</server>
<name>
Variable using illegal function in expansion
</name>
<command>
http://%HOSTIP:%HTTPPORT/%TESTNUMBER --variable what=hello --expand-data "--{{what:trim:super}}"
</command>
</client>
#
# Verify data after the test has been "shot"
<verify>
<errorcode>
2
</errorcode>
</verify>
</testcase>

33
tests/data/test453 Normal file
View File

@ -0,0 +1,33 @@
<testcase>
<info>
<keywords>
HTTP
variables
</keywords>
</info>
#
# Client-side
<client>
<server>
http
</server>
<name>
Variable output containing null byte
</name>
<file name="%LOGDIR/junk">
%hex[%01%02%03%00%04%05%06]hex%
</file>
<command>
http://%HOSTIP:%HTTPPORT/%TESTNUMBER --variable what@%LOGDIR/junk --expand-data "{{what}}"
</command>
</client>
#
# Verify data after the test has been "shot"
<verify>
<errorcode>
2
</errorcode>
</verify>
</testcase>

34
tests/data/test454 Normal file
View File

@ -0,0 +1,34 @@
<testcase>
<info>
<keywords>
variables
</keywords>
</info>
#
# Server-side
<reply>
</reply>
#
# Client-side
<client>
<server>
http
</server>
<name>
Variable using illegal function separator
</name>
<command>
http://%HOSTIP:%HTTPPORT/%TESTNUMBER --variable what=hello --expand-data "--{{what:trim,url}}"
</command>
</client>
#
# Verify data after the test has been "shot"
<verify>
<errorcode>
2
</errorcode>
</verify>
</testcase>

52
tests/data/test455 Normal file
View File

@ -0,0 +1,52 @@
<testcase>
<info>
<keywords>
variables
</keywords>
</info>
#
# Server-side
<reply>
<data crlf="yes">
HTTP/1.1 200 OK
Date: Tue, 09 Nov 2010 14:49:00 GMT
Server: test-server/fake
Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
ETag: "21025-dc7-39462498"
Accept-Ranges: bytes
Content-Length: 6
Connection: close
Content-Type: text/html
Funny-head: yesyes
-foo-
</data>
</reply>
#
# Client-side
<client>
<server>
http
</server>
<name>
Variable using base64
</name>
<command>
--variable moby="Call me Ishmael" --expand-url "http://%HOSTIP:%HTTPPORT/{{moby:b64}}/%TESTNUMBER"
</command>
</client>
#
# Verify data after the test has been "shot"
<verify>
<protocol crlf="yes">
GET /%b64[Call me Ishmael]b64%/%TESTNUMBER HTTP/1.1
Host: %HOSTIP:%HTTPPORT
User-Agent: curl/%VERSION
Accept: */*
</protocol>
</verify>
</testcase>

5
winbuild/MakefileBuild.vc
View File

@ -666,7 +666,8 @@ CURL_FROM_LIBCURL=$(CURL_DIROBJ)\tool_hugehelp.obj \
$(CURL_DIROBJ)\warnless.obj \
$(CURL_DIROBJ)\curl_multibyte.obj \
$(CURL_DIROBJ)\version_win32.obj \
$(CURL_DIROBJ)\dynbuf.obj
$(CURL_DIROBJ)\dynbuf.obj \
$(CURL_DIROBJ)\base64.obj
$(PROGRAM_NAME): $(CURL_DIROBJ) $(CURL_FROM_LIBCURL) $(EXE_OBJS)
$(CURL_LINK) $(CURL_LFLAGS) $(CURL_LIBCURL_LIBNAME) $(WIN_LIBS) $(CURL_FROM_LIBCURL) $(EXE_OBJS)
@ -689,6 +690,8 @@ $(CURL_DIROBJ)\version_win32.obj: ../lib/version_win32.c
$(CURL_CC) $(CURL_CFLAGS) /Fo"$@" ../lib/version_win32.c
$(CURL_DIROBJ)\dynbuf.obj: ../lib/dynbuf.c
$(CURL_CC) $(CURL_CFLAGS) /Fo"$@" ../lib/dynbuf.c
$(CURL_DIROBJ)\base64.obj: ../lib/base64.c
$(CURL_CC) $(CURL_CFLAGS) /Fo"$@" ../lib/base64.c
$(CURL_DIROBJ)\curl.res: $(CURL_SRC_DIR)\curl.rc
rc $(CURL_RC_FLAGS)