Skip to main content

v0.0.128

· 3 min read
Shourya Shashank
Shourya Shashank
Maintainer of Predacons

  • New Features

    • Introduced streaming capabilities for text and chat generation, allowing real-time output.
    • Added new functions: text_stream and chat_stream for enhanced streaming functionality.

  • Bug Fixes

  • Updated error handling for streaming setup and generation processes with descriptive messages. -Examples
# %%
import predacons

# %%
predacons.rollout()

# %%
import os
os.environ['CUDA_VISIBLE_DEVICES'] ='0'
import torch
if torch.cuda.is_available():
    # torch.set_default_device('cuda')
    print("Using GPU")
else:
    print("No GPU available")

# %%
model_path = "Predacon/Pico-Lamma-3.2-1B-Reasoning-Instruct"

# %%
model = predacons.load_model(model_path)
tokenizer = predacons.load_tokenizer(model_path)

# %%
seq = "The quick brown fox jumps over the"

# %%


thread,stream = predacons.text_generate(model=model, tokenizer = tokenizer, sequence = seq, max_length=100, temperature=0.1,stream=True)

# %%
thread.start()
try:
    out = ""
    for new_text in stream:
        out = out + new_text
        print(new_text, end=" ")
finally:
    thread.join()

# %%
a = predacons.text_stream(model=model, tokenizer = tokenizer, sequence = seq, max_length=100, temperature=0.1)

# %%
a

# %%
chat = [
    {"role": "user", "content": "A train travelling at a speed of 60 km/hr is stopped in 15 seconds by applying the brakes. Determine its retardation."},
]

# %%
thread,stream = predacons.chat_generate(model=model, tokenizer = tokenizer, sequence = chat, max_length=500, temperature=0.1,stream=True)

# %%
thread.start()
try:
    out = ""
    for new_text in stream:
        out = out + new_text
        print(new_text, end="")
finally:
    thread.join()

# %%
b = predacons.chat_stream(model=model, tokenizer = tokenizer, sequence = chat, max_length=500, temperature=0.1)

# %%
b

  • Documentation

    • Enhanced documentation for functions to include detailed parameter descriptions and functionalities.

  • Chores

    • Incremented version number to 0.0.128 and updated package dependencies, removing compatibility concerns.

Walkthrough​

This pull request introduces significant enhancements to the Generate class and the predacons module by adding streaming capabilities for output generation. New methods are implemented to allow real-time streaming of both text and chat outputs. The rollout function in predacons.py is updated with a new version number and additional print statements for better documentation. The setup.py file reflects a version increment and removal of specific dependencies, indicating a shift in package requirements.

Changes​

FileChange Summary
app/predacons/src/generate.py- Added methods for streaming outputs: generate_output_stream, generate_chat_output_stream, generate_output_from_model_stream, generate_chat_output_from_model_stream.
  • Restructured __generate_chat_output to support streaming and updated error handling. | | app/predacons/src/predacons.py | - Updated rollout function version from v0.0.126 to v0.0.128.
  • Added new functions: text_stream, chat_stream.
  • Updated generate, text_generate, and chat_generate functions to include a stream parameter. | | setup.py | - Updated version from 0.0.126 to 0.0.128.
  • Removed dependencies for torch and bitsandbytes from install_requires. | | app/predacons/init.py | - Added text_stream and chat_stream to the list of exported functions. |

What's Changed​

Full Changelog: https://github.com/Predacons/predacons/compare/v0.0.126...v0.0.128

Github Release Page: https://github.com/Predacons/predacons/releases/tag/v0.0.128