This is tutorial for Visual Studio C++ how to use resdecode and resconvert in programs.
To execute resdecode.exe or resconvert.exe you need to use ShellExecute function.
Here's a code i use:

std::string resdecode = path + "resdecode.exe";
std::string resconvert = path + "resconvert.exe";
ShellExecute(NULL, "open", resdecode.c_str(), "rpk name", path.c_str(), 0);
ShellExecute(NULL, "open", resconvert.c_str(), "rdb name", path.c_str(), 0);
std::string path is a path to the program exe file.

Notes:
resdecode.c_str() / resconvert.c_str() - a location of resdecode.exe/resconvert.exe
path.c_str() - is a path where the .rpk/.rdb file is

More about ShellExecute function you can read here: Link
Exe path can be determined with this function:

std::string ExePath() {
	char buffer[MAX_PATH];
	GetModuleFileName(NULL, buffer, MAX_PATH);
	std::string::size_type pos = std::string(buffer).find_last_of("\\/");
	return std::string(buffer).substr(0, pos+1);
}

If you have problems with this code, try changing Character set from Unicode to ASCII in your Visual Studio settings.